Communication terminal, image management apparatus, image processing system, method for controlling display, and computer program product

ABSTRACT

A communication terminal includes circuitry. The circuitry is configured to calculate, for each of a plurality of captured-image data items that are obtained by continuous image capturing, a center point of an object detection area of the captured-image data item. The circuitry is further configured to render a heat map image having a gradation of color, the gradation of color of the heat map image being determined based on weighting of colors of the heat map image with the calculated center points of the captured-image data items. The circuitry is further configured to map the heat map image on a captured-image represented by one of the plurality of captured-image data items. The circuitry is further configured to display the captured-image on which the heat map image is mapped.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is based on and claims priority pursuant to 35USC § 119(a) to Japanese Patent Application Nos. 2016-043742, filed onMar. 7, 2016, 2016-068644 filed on Mar. 30, 2016, 2016-072662 filed onMar. 31, 2016, and 2017-033507, filed on Feb. 24, 2017 in the JapanPatent Office, the entire disclosures of which are hereby incorporatedby reference herein.

BACKGROUND Technical Field

The present disclosure relates to a communication terminal, an imagemanagement apparatus, an image processing system, a method forcontrolling display of captured images, and a computer program product.

Description of the Related Art

Monitoring systems that include a monitoring camera and an imageprocessing function are known. Persons' stay or a track of persons isanalyzed using the monitoring system. The analysis results are used fora store management, a merchandise planning, the construction andimprovement of roads, the prevention of crime, etc.

Further, a system is known that indicates a person's stay or a person'strack using a heat map image separately from a captured image containingpersons. Further, analysis information that is obtained by analyzingimage data is edited using a computer to make it easier for a user tounderstand the analysis information. For example, bar charts or circlegraphs are used when comparing plural types of analysis information.Furthermore, the analysis information is sometimes indicated indifferent to enable a user to visually recognize the analysisinformation or to emphasis the analysis information.

SUMMARY

A communication terminal includes circuitry. The circuitry is configuredto calculate, for each of a plurality of captured-image data items thatare obtained by continuous image capturing, a center point of an objectdetection area of the captured-image data item. The circuitry is furtherconfigured to render a heat map image having a gradation of color, thegradation of color of the heat map image being determined based onweighting of colors of the heat map image with the calculated centerpoints of the captured-image data items. The circuitry is furtherconfigured to map the heat map image on a captured-image represented byone of the plurality of captured-image data items. The circuitry isfurther configured to display the captured-image on which the heat mapimage is mapped.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the embodiments and many of theattendant advantages and features thereof can be readily obtained andunderstood from the following detailed description with reference to theaccompanying drawings, wherein:

FIG. 1A is a left side view of an image capturing device according to anembodiment of the present invention;

FIG. 1B is a front view of the image capturing device of FIG. 1A;

FIG. 1C is a plan view of the image capturing device of FIG. 1A;

FIG. 2 is an illustration for explaining how a user uses the imagecapturing device of FIG. 1A to 1C according to an embodiment of thepresent invention;

FIG. 3A is a view illustrating a front side of a hemispherical imagecaptured by the image capturing device of FIG. 1 according to anembodiment of the present invention;

FIG. 3B is a view illustrating a back side of the hemispherical imagecaptured by the image capturing device of FIG. 1 according to anembodiment of the present invention;

FIG. 3C is a view illustrating an image captured by the image capturingdevice of FIG. 1, represented by Mercator projection according to anembodiment of the present invention;

FIG. 4A is an illustration for explaining how the image represented byMercator projection covers a surface of a sphere according to anembodiment of the present invention;

FIG. 4B is a view illustrating a full spherical panoramic imageaccording to an embodiment of the present invention;

FIG. 5 is a view illustrating positions of a virtual camera and apredetermined area in a case in which the full spherical panoramic imageis represented as a three-dimensional solid sphere according to anembodiment of the present invention;

FIG. 6A is a perspective view of FIG. 5;

FIG. 6B is a view illustrating an image of the predetermined area on adisplay of a communication terminal according to an embodiment of thepresent invention;

FIG. 7 is a view illustrating a relation between predetermined-areainformation and a predetermined-area image according to an embodiment ofthe present invention;

FIG. 8 is a schematic view illustrating an image communication systemaccording to an embodiment of the present invention;

FIG. 9 is a block diagram illustrating a hardware configuration of theimage capturing device of FIG. 1 according to an embodiment of thepresent invention;

FIG. 10 is a block diagram illustrating a hardware configuration of acommunication terminal 3 of FIG. 8 according to an embodiment of thepresent invention;

FIG. 11 is a block diagram illustrating a hardware configuration of anyone of the image management system and a communication terminal 7 ofFIG. 8 according to an embodiment of the present invention;

FIG. 12 is a schematic diagram illustrating operation performed by theimage communication system according to an embodiment of the presentinvention;

FIG. 13 is a block diagram illustrating a functional configuration ofthe image communication system according to an embodiment of the presentinvention;

FIG. 14 is an example of a person image management table according to anembodiment of the present invention;

FIG. 15 is an illustration for explaining a position and area of aperson image according to an embodiment of the present invention;

FIG. 16A is an illustration for explaining a score table according to anembodiment of the present invention;

FIG. 16B is an illustration for explaining a drawing point tableaccording to an embodiment of the present invention;

FIG. 17 is a flowchart illustrating an operation of generating a heatmap image according to a first embodiment of the present invention;

FIG. 18 is a flowchart illustrating an operation of generating the heatmap image according to a first embodiment of the present invention;

FIGS. 19A to 19C are illustrations for explaining an operation ofdetermining a drawing point (a first center point of a first persondetection area and a score of the first center point) according to anembodiment of the present invention;

FIGS. 20A to 20C are illustrations for explaining an operation ofdetermining a drawing point (a second center point of a second persondetection area and a score of the second center point) according to anembodiment of the present invention;

FIGS. 21A and 21B are illustrations for explaining an operation ofdetermining a drawing point (an average center point of the first centerpoint and the second center point, and a score of the average centerpoint) according to an embodiment of the present invention;

FIGS. 22A to 22C are illustrations for explaining an operation ofdetermining a drawing point (a third center point of a third persondetection area and a score of the third center point) according to anembodiment of the present invention;

FIGS. 23A and 23B are illustrations for describing an operation ofdetermining a drawing point (the average center point of the firstcenter point and the second center point, and the score thereof), and adrawing point (the average center point of the second center point andthe third center point, and a score thereof) according to an embodimentof the present invention;

FIGS. 24A and 24B are illustrations for explaining an operation ofdetermining a drawing point (an average center point of a plurality ofaverage center points (a second average center point) and a scorethereof) according to an embodiment of the present invention;

FIGS. 25A to 25C are illustrations for explaining an operation ofdetermining a drawing point (a fourth center point of a fourth persondetection area and a score of the fourth center point) according to anembodiment of the present invention;

FIGS. 26A to 26C are illustrations for explaining the drawing points(various types of center points in each of a plurality of images and thescores of the center points) according to an embodiment of the presentinvention;

FIGS. 27A and 27B are illustrations for explaining integration of thedrawing points in a specific range in a plurality of images according toan embodiment of the present invention;

FIG. 28 is a view illustrating the predetermined-area image to which theheat map image generated based on the drawing points is attachedaccording to an embodiment of the present invention;

FIG. 29 is a view illustrating the full spherical panoramic image towhich the heat map image generated based on the drawing points isattached according to an embodiment of the present invention;

FIG. 30 is an illustration for describing a relation between the fullspherical panoramic image and the heat map image according to anembodiment of the present invention;

FIG. 31 is a flowchart illustrating an operation of generating the heatmap image according to a second embodiment of the present invention;

FIG. 32 is a flowchart illustrating an operation of generating the heatmap image according to a second embodiment of the present invention;

FIGS. 33A and 33B are illustrations for describing an operation ofdetermining each drawing point (the first to fourth center points of thefirst to fourth person detection areas and the score of each of thefirst to fourth center points) according to an embodiment of the presentinvention;

FIGS. 34A and 34B are illustrations for describing an operation ofdetermining a drawing point (an average center point of the first centerpoint and the second center point, and the score thereof), a drawingpoint (an average center point of the first center point and the thirdcenter point, and the score thereof), and an drawing point (an averagecenter point of the second center point and the third center point, andthe score thereof) according to an embodiment of the present invention;

FIG. 35 is a schematic diagram illustrating operation performed by theimage communication system according to a third embodiment of thepresent invention;

FIG. 36 is a functional block diagram of the image communication systemaccording to a third embodiment of the present invention;

FIG. 37 is an example of a person image management table according to athird embodiment of the present invention;

FIG. 38 is an illustration for explaining a position and area of aperson image according to a third embodiment of the present invention;

FIG. 39 is a flowchart illustrating an operation of generating personimage information according to a third embodiment of the presentinvention;

FIG. 40 is a plan view illustrating an example situation where the fullspherical panoramic image is captured by the image capturing deviceaccording to a third embodiment of the present invention;

FIG. 41 is a side view illustrating an example situation where the fullspherical panoramic image is captured by the image capturing deviceaccording to a third embodiment of the present invention;

FIG. 42 is an illustration for explaining calculation of a diameter of ahead (face) according to a third embodiment of the present invention;

FIG. 43 is an illustration for explaining calculation of a distancebetween a center of a head (face) and a foot according to a thirdembodiment of the present invention;

FIG. 44 is an illustration for explaining calculation of an anglebetween a center of a head and a horizontal line according to a thirdembodiment of the present invention;

FIG. 45 is an illustration for explaining calculation of an anglebetween a foot and a horizontal line according to a third embodiment ofthe present invention;

FIG. 46A is a view illustrating the predetermined-area image accordingto a third embodiment of the present invention;

FIG. 46B is a view illustrating the predetermined-area image on whichthe heat map is superimposed according to a third embodiment of thepresent invention;

FIG. 47A is a view illustrating the predetermined-area image accordingto a related art of a third embodiment of the present invention;

FIG. 47B is a view illustrating the predetermined-area image on whichthe heat map is superimposed according to a related art of a thirdembodiment of the present invention;

FIG. 48 is a schematic diagram illustrating operation performed by theimage communication system according to a fourth embodiment of thepresent invention;

FIG. 49 is a functional block diagram of the image communication systemaccording to a fourth embodiment of the present invention;

FIG. 50 is a flowchart illustrating an operation of generating the heatmap image according to a fourth embodiment of the present invention;

FIG. 51 is a flowchart illustrating an operation of generating the heatmap image according to a fourth embodiment of the present invention;

FIGS. 52A to 52D are illustrations for explaining calculation of a staypoint according to a fourth embodiment of the present invention;

FIGS. 53A to 53D are illustrations for explaining calculation of a staypoint according to a fourth embodiment of the present invention;

FIGS. 54A to 54D are illustrations for explaining calculation of a staypoint according to a fourth embodiment of the present invention;

FIG. 55 is an illustration for explaining calculation of a stay pointaccording to a fourth embodiment of the present invention;

FIG. 56 is a view illustrating an example of the heat map imageaccording to a fourth embodiment of the present invention;

FIG. 57 is a view illustrating an example of an analysis informationscreen displayed by the communication terminal according to a fifthembodiment of the present invention;

FIG. 58 is a block diagram illustrating a functional configuration ofthe image communication system according to a fifth embodiment of thepresent invention;

FIG. 59 is an illustration for explaining drawing of a bar chartaccording to a fifth embodiment of the present invention;

FIG. 60 is a flowchart illustrating an operation of generating the barchart according to a fifth embodiment of the present invention;

FIGS. 61A and 61B are illustrations for explaining rendering of the barchart according to a fifth embodiment of the present invention;

FIG. 62 is a view illustrating an example of a setting screen displayedon a display of the communication terminal according to a fifthembodiment of the present invention;

FIGS. 63A and 63B are example illustrations for explaining the relationbetween a full sphere and the position and angle of view of the virtualcamera according to a fifth embodiment of the present invention;

FIGS. 64A to 64E are illustrations for explaining switching performedwhen the image data is attached to an inner surface or an outer surfaceof a full sphere according to a fifth embodiment of the presentinvention;

FIGS. 65A and 65B are views illustrating the full spherical image whichis smoothed based on a height of the bar chart according to a fifthembodiment of the present invention, and

FIG. 66 is an illustration for explaining smoothing of the height of barchart in each area based on the height of the bar chart according to afifth embodiment of the present invention.

The accompanying drawings are intended to depict embodiments of thepresent invention and should not be interpreted to limit the scopethereof. The accompanying drawings are not to be considered as drawn toscale unless explicitly noted.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specificterminology is employed for the sake of clarity. However, the disclosureof this specification is not intended to be limited to the specificterminology so selected and it is to be understood that each specificelement includes all technical equivalents that have a similar function,operate in a similar manner, and achieve a similar result.

As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the multiple forms as well, unless the context clearly indicatesotherwise.

First Embodiment

A first embodiment of the present invention will be describedhereinafter with reference to drawings.

First, a description is given of an operation of generating a fullspherical panoramic image, with reference to FIGS. 1 to 7.

First, a description is given of an external view of an image capturingdevice 1 with reference to FIGS. 1A to 1C. The image capturing device 1is a digital camera for acquiring captured images from which a360-degree full spherical panoramic image is generated. FIGS. 1A to 1Care respectively a left side view, a front view, and a plan view of theimage capturing device 1.

As illustrated in FIG. 1A, the image capturing device 1 has a shape suchthat one can hold it with one hand. Further, as illustrated in FIGS. 1Ato 1C, an imaging element 103 a is provided on a front side (anteriorside) of an upper section of the image capturing device 1, and animaging element 103 b is provided on a back side (rear side) thereof.Furthermore, as illustrated in FIG. 1B, an operation unit 115 such as ashutter button is provided on the back side (rear side) of the imagecapturing device 1.

Hereinafter, a description is given of a situation where the imagecapturing device 1 is used with reference to FIG. 2. FIG. 2 is anexample illustration for explaining how a user uses the image capturingdevice 1. As illustrated in FIG. 2, for example, the image capturingdevice 1 is used for capturing objects surrounding the user who isholding the image capturing device 1 in his/her hand. The imagingelements 103 a and 103 b illustrated in FIGS. 1A to 1C capture theobjects surrounding the user to obtain two hemispherical images.

Hereinafter, a description is given of an overview of an operation ofgenerating the full spherical panoramic image from the image captured bythe image capturing device 1 with reference to FIGS. 3A to 3C and FIGS.4A and 4B. FIG. 3A is a view illustrating a front side of ahemispherical image captured by the image capturing device 1. FIG. 3B isa view illustrating a back side of the hemispherical image captured bythe image capturing device 1. FIG. 3C is a view illustrating an imagerepresented by Mercator projection. The image represented by Mercatorprojection as illustrated in FIG. 3C is referred to as a “Mercatorimage” hereinafter. FIG. 4A is an illustration for explaining how theMercator image covers a surface of a sphere. FIG. 4B is a viewillustrating the full spherical panoramic image.

As illustrated in FIG. 3A, the image captured by the imaging element 103a is a curved hemispherical image (front side) taken through a fisheyelens 102 a (FIG. 9). Also, as illustrated in FIG. 3B, an image capturedby the imaging element 103 b is a curved hemispherical image (back side)taken through a fisheye lens 102 b (FIG. 9). The image capturing device1 combines the hemispherical image (front side) and the hemisphericalimage (back side), which is reversed by 180-degree from each other, togenerate the Mercator image as illustrated in FIG. 3C.

The Mercator image is pasted on the sphere surface using Open GraphicsLibrary for Embedded Systems (OpenGL ES) as illustrated in FIG. 4A.Thus, the full spherical panoramic image as illustrated in FIG. 4B isgenerated. In other words, the full spherical panoramic image isrepresented as the Mercator image facing toward a center of the sphere.It should be noted that OpenGL ES is a graphic library used forvisualizing two-dimensional (2D) and three-dimensional (3D) data. Thefull spherical panoramic image is either a still image or a movie.

One may feel strange as viewing the full spherical panoramic image,because the full spherical panoramic image is an image attached to thesphere surface. To resolve this strange feeling, an image of apredetermined area, which is a part of the full spherical panoramicimage, is displayed as a planar image having less curves. The image ofthe predetermined area is referred to as a “predetermined-area image”hereinafter. Hereinafter, a description is given of displaying thepredetermined-area image with reference to FIG. 5 and FIGS. 6A and 6B.

FIG. 5 is a view illustrating positions of a virtual camera IC and apredetermined area T in a case in which the full spherical panoramicimage is represented as a three-dimensional solid sphere. The virtualcamera IC corresponds to a position of a point of view of a user who isviewing the full spherical panoramic image represented as thethree-dimensional solid sphere. FIG. 6A is a perspective view of FIG. 5.FIG. 6B is a view illustrating the predetermined-area image displayed ona display. In FIG. 6A, the full spherical panoramic image illustrated inFIG. 4B is represented as a three-dimensional solid sphere CS. Assumingthat the generated full spherical panoramic image is the solid sphereCS, the virtual camera IC is outside of the full spherical panoramicimage as illustrated in FIG. 5. The predetermined area T in the fullspherical panoramic image is specified by predetermined-area informationof the position of the virtual camera IC in the full spherical panoramicimage. This predetermined-area information is represented by acoordinate x (rH), a coordinate y (rV), and an angle of view α (angle),for example. Zooming of the predetermined area T is implemented byenlarging or reducing a range of the angle of view α. In other words,zooming of the predetermined area T is implemented by enlarging orreducing an arc. Further, zooming of the predetermined area T isimplemented by moving the virtual camera IC toward or away from the fullspherical panoramic image.

An image of the predetermined area T in the full spherical panoramicimage illustrated in FIG. 6A is displayed on a display as thepredetermined-area image, as illustrated in FIG. 6B. FIG. 6B illustratesan image represented by the predetermined-area information (x, y, α),which is set by default.

Hereinafter, a description is given of a relation between thepredetermined-area information and the predetermined-area image withreference to FIG. 7. FIG. 7 is a view illustrating a relation betweenthe predetermined-area information and the predetermined-area image. Asillustrated in FIG. 7, a center point CP of 2L provides the parameters(x, y) of the predetermined-area information, where 2L denotes adiagonal angle of view of the predetermined area T represented by anangle of view α of the virtual camera IC. Distance f denotes a distancefrom the virtual camera IC to the central point CP. In FIG. 7, atrigonometric function equation generally expressed by the followingequation (1) is satisfied.Lf=tan(α/2)  (Equation 1)

Hereinafter, a description is given of an overview of a configuration ofan image communication system 10 according to this embodiment withreference to FIG. 8.

FIG. 8 is a schematic diagram illustrating a configuration of the imagecommunication system 10 according to this embodiment.

As illustrated in FIG. 8, the image communication system 10 according tothe present embodiment includes the image capturing device 1, acommunication terminal 3, a wireless router 9 a, an image managementsystem 5, and a communication terminal 7.

As described above, the image capturing device 1 is a digital cameracapable of obtaining the full spherical panoramic image. Alternatively,the image capturing device 1 may be a typical digital camera. In a casein which the communication terminal 3 includes a camera, thecommunication terminal 3 may also operate as the digital camera. In thisembodiment, a description is given of a case in which the imagecapturing device 1 is a digital camera that is capable of obtaining thefull spherical panoramic image, in order to make the description simple.In this example, the communication terminal 3 operates at least as adocking station that charges the image capturing device 1 or exchangesdata with the image capturing device 1. The communication terminal 3communicates data with the image capturing device 1 via a contact. Inaddition, the communication terminal 3 communicates data with the imagemanagement system 5 via the wireless router 9 a and a communicationnetwork 9. The communication network 9 is implemented by, for example,the Internet.

The image management system 5 communicates data with the communicationterminal 3 and the communication terminal 7 via the communicationnetwork 9. The image management system 5 is implemented by, for example,a server computer. The image management system 5 is installed withOpenGL ES to generate the full spherical panoramic image.

The communication terminal 7 communicates data with the image managementsystem 5 via the communication network 9. The communication terminal 7is implemented by, for example, a laptop computer. It should be notedthat the image management system 5 may be implemented by either a singleserver computer or a plurality of server computers.

Hereinafter, a description is given of hardware configurations of theimage capturing device 1, the communication terminal 3, thecommunication terminal 7, and the image management system 5 according tothis embodiment with reference to FIGS. 9 to 11.

First, a description is given of a hardware configuration of the imagecapturing device 1 with reference to FIG. 9. FIG. 9 is a block diagramillustrating a hardware configuration of the image capturing device 1.Although a description is given of a case in which the image capturingdevice 1 is an omnidirectional image capturing device having two imagingelements, the image capturing device 1 may include three or more imagingelements. In addition, the image capturing device 1 is not necessarilyan image capturing device dedicated to omnidirectional image capturing.Alternatively, an external omnidirectional image capturing unit may beattached to a general digital camera or smartphone to implement an imagecapturing device having a substantially same function as that of theimage capturing device 1.

As illustrated in FIG. 9, the image capturing device 1 includes animaging unit 101, an image processor 104, an imaging controller 105, amicrophone 108, a sound processor 109, a central processing unit (CPU)111, a read only memory (ROM) 112, a static random access memory (SRAM)113, a dynamic random access memory (DRAM) 114, an operation unit 115, anetwork interface (I/F) 116, a communication unit 117, and an antenna117 a.

The imaging unit 101 includes two wide-angle lenses (so-called fish-eyelenses) 102 a and 102 b, each having an angle of view of equal to orgreater than 180 degrees so as to form a hemispheric image. The imagingunit 101 further includes the two imaging elements 103 a and 103 bcorresponding to the wide-angle lenses 102 a and 102 b respectively. Theimaging elements 103 a and 103 b each includes an image sensor such as acomplementary metal oxide semiconductor (CMOS) sensor and acharge-coupled device (CCD) sensor, a timing generation circuit, and agroup of registers. The image sensor converts an optical image formed bythe wide-angle lenses 102 a and 102 b into electric signals to outputimage data. The timing generation circuit generates horizontal orvertical synchronization signals, pixel clocks and the like for theimage sensor. Various commands, parameters and the like for operationsof the imaging elements 103 a and 103 b are set in the group ofregisters.

Each of the imaging elements 103 a and 103 b of the imaging unit 101 isconnected to the image processor 104 via a parallel I/F bus. Inaddition, each of the imaging elements 103 a and 103 b of the imagingunit 101 is connected to the imaging controller 105 via a serial I/F bussuch as an I2C bus. The image processor 104 and the imaging controller105 are each connected to the CPU 111 via a bus 110. Furthermore, theROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the networkI/F 116, the communication unit 117, and the electronic compass 118 arealso connected to the bus 110.

The image processor 104 acquires the image data from each of the imagingelements 103 a and 103 b via the parallel I/F bus and performspredetermined processing on each image data. Thereafter, the imageprocessor 104 combines these image data to generate data of the Mercatorimage as illustrated in FIG. 3C.

The imaging controller 105 usually functions as a master device whilethe imaging elements 103 a and 103 b each usually functions as a slavedevice. The imaging controller 105 sets commands and the like in thegroup of registers of the imaging elements 103 a and 103 b via the I2Cbus. The imaging controller 105 receives necessary commands and the likefrom the CPU 111. Further, the imaging controller 105 acquires statusdata and the like of the group of registers of the imaging elements 103a and 103 b via the I2C bus. The imaging controller 105 sends theacquired status data and the like to the CPU 111.

The imaging controller 105 instructs the imaging elements 103 a and 103b to output the image data at a time when the shutter button of theoperation unit 115 is pressed. The image capturing device 1 may have apreview function or support displaying movie. In this case, the imagedata are continuously output from the imaging elements 103 a and 103 bat a predetermined frame rate (frames per minute).

Furthermore, the imaging controller 105 operates in cooperation with theCPU 111 to synchronize times when the imaging elements 103 a and 103 boutput the image data. It should be noted that although the imagecapturing device 1 does not include a display in this embodiment,alternatively the image capturing device 1 may include the display.

The microphone 108 converts sounds to audio data (signal). The soundprocessor 109 acquires the audio data from the microphone 108 via an I/Fbus and performs predetermined processing on the audio data.

The CPU 111 controls an entire operation of the image capturing device 1and performs necessary processing. The ROM 112 stores various programsfor the CPU 111. The SRAM 113 and the DRAM 114 each operates as a workmemory to store the program loaded from the ROM 112 for execution by theCPU 111 or data in current processing. More specifically, the DRAM 114stores the image data currently processed by the image processor 104 andthe data of the Mercator image on which processing has been performed.

The operation unit 115 collectively refers to various operation keys, apower switch, the shutter button, and a touch panel having functions ofboth displaying information and receiving input from a user, which maybe used in combination. The user operates the operation keys to inputvarious photographing modes or photographing conditions.

The network I/F 116 collectively refers to an interface circuit such asan universal serial bus (USB) I/F that allows the image capturing device1 to communicate data with an external media such as a SD card or anexternal personal computer. The network I/F 116 supports at least one ofwired and wireless communications. The data of the Mercator image, whichis stored in the DRAM 114, is stored in the external media via thenetwork I/F 116 or transmitted to the external device such as thecommunication terminal 3 via the network I/F 116.

The communication unit 117 is implemented by, for example, an interfacecircuit. The communication unit 117 communicates data with an externaldevice such as the communication terminal 3 via the antenna 117 a by anear distance wireless communication such as Wi-Fi and Near FieldCommunication (NFC). The communication unit 117 is also capable oftransmitting the data of Mercator image to the external device such asthe communication terminal 3.

The electronic compass 118 calculates an orientation and a tilt (rollangle) of the image capturing device 1 from the Earth's magnetism tooutput orientation and tilt information. This orientation and tiltinformation is an example of related information, which is meta datadescribed in compliance with Exif. This information is used for imageprocessing such as image correction of the captured image. Further, therelated information also includes a date and time when the image iscaptured by the image capturing device 1, and a size of the image data.

Hereinafter, a description is given of a hardware configuration of thecommunication terminal 3 with reference to FIG. 10. FIG. 10 is a blockdiagram illustrating a hardware configuration of the communicationterminal 3, which in this embodiment is implemented by a docking stationhaving a wireless communication capability.

As illustrated in FIG. 10, the communication terminal 3 includes a CPU301, a ROM 302, a RAM 303, an electrically erasable programmable ROM(EEPROM) 304, and a CMOS sensor 305. The CPU 301 controls entireoperation of the communication terminal 3. The ROM 302 stores basicinput/output programs. The CPU 301 uses the RAM 303 as a work area whenexecuting programs or processing data. The EEPROM 304 performs datareading and writing under control of the CPU 301. The CMOS sensor 305 isan imaging element that captures an image of an object to obtain imagedata under control of the CPU 301.

The EEPROM 304 stores an operating system (OS) for execution by the CPU301, other programs, and various data. Instead of the CMOS sensor 305, aCCD sensor may be used.

Further, the communication terminal 3 includes an antenna 313 a, acommunication unit 313, a global positioning systems (GPS) receiver 314,and a bus line 310. The communication unit 313, which is implemented by,for example, an interface circuit, communicates data with otherapparatuses or terminals by wireless communication signals using theantenna 313 a. The GPS receiver 314 receives GPS signals containingposition information of the communication terminal 3 with GPS satellitesor an indoor Messaging system as indoor GPS. This position informationof communication terminal 3 is represented by, for example, a latitude,longitude, and altitude. The bus line 310 electrically connects thoseparts or devices of the communication terminal 3 to each other. Examplesof the bus line 310 include an address bus and a data bus.

Hereinafter, a description is given of hardware configurations of theimage management system 5 and the communication terminal 7, which inthis embodiment is implemented by a laptop computer, with reference toFIG. 11. FIG. 11 is a block diagram illustrating a hardwareconfiguration of any one of the image management system 5 and thecommunication terminal 7. In this embodiment, both the image managementsystem 5 and the communication terminal 7 are implemented by a computer.Therefore, a description is given of a configuration of the imagemanagement system 5, and the description of a configuration of thecommunication terminal 7 is omitted, having the same or substantiallysame configuration as that of the image management system 5.

The image management system 5 includes a CPU 501, a ROM 502, a RAM 503,an HD 504, a hard disc drive (HDD) 505, a media drive 507, a display508, a network I/F 509, a keyboard 511, a mouse 512, a compact-disc readonly memory (CD-ROM) drive 514, and a bus line 510. The CPU 501 controlsentire operation of the image management system 5. The ROM 502 storesprograms such as an initial program loader to boot the CPU 501. The CPU501 uses the RAM 503 as a work area when executing programs orprocessing data. The HD 504 stores various data such as programs for theimage management system 5. The HDD 505 controls reading and writing ofdata from and to the HD 504 under control of the CPU 501. The mediadrive 507 controls reading and writing (storing) of data from and to arecording medium 506 such as a flash memory. The display 508 displaysvarious information such as a cursor, menus, windows, characters, orimages. The network I/F 509 communicates data with another apparatussuch as the communication terminal 3 and the communication terminal 7via the communication network 9. The keyboard 511 includes a pluralityof keys to allow a user to input characters, numbers, and variousinstructions. The mouse 512 allows a user to input an instruction forselecting and executing various functions, selecting an item to beprocessed, or moving the cursor. The CD-ROM drive 514 controls readingand writing of data from and to a CD-ROM 513 as an example of aremovable recording medium. The bus line 510 electrically connects thoseparts or devices of the image management system 5 to each other asillustrated in FIG. 14. Examples of the bus line 510 include an addressbus and a data bus.

Hereinafter, a description is given of an overview of processingperformed by the image communication system 10 according to the presentembodiment with reference to FIG. 12. FIG. 12 is a schematic diagramillustrating operation performed by the image communication system 10according to the present embodiment.

First, the communication terminal 3 acquires the captured-image data,the predetermined-area information, and the related information from theimage capturing device 1 (S1). The communication terminal 3 transmitsthe captured-image data, the predetermined-area image, and the relatedinformation to the image management system 5 (S2). For example, thecommunication terminal 3 performs this data transmission at one-minuteintervals. Next, the image management system 5 detects a person image,which is an image of a person, from the captured-image data (S3). Theimage management system 5 transmits the predetermined-area image and therelated information (S4). Next, in response to a request from thecommunication terminal 7, the image management system 5 transmits ascore table and a drawing point table, person detection information, anda group of captured-image data items (S5). The drawing point tabletransmitted at S5 contains blank data in cells for positions of drawingpoints and scores. Thus, the communication terminal 7 stores, in amemory 7000 (described later), the score table, the drawing point table(containing blank data in the cells for a point of drawing point and ascore), the person detection information, and the group ofcaptured-image data items. A captured-image ID for identifying thecaptured-image data is attached to each of the captured-image dataitems.

Next, the communication terminal 7 generates a heat map image asillustrated in FIGS. 28 and 29 (S6). This enables a viewer Y torecognize the movement of persons even when the viewer Y does not view aplurality of full spherical panoramic images (or predetermined-areaimages) for a specific period of time.

Hereinafter, a description is given of a functional configuration of theimage communication system 10 according to this embodiment withreference to FIGS. 9 to 11 and 13. FIG. 13 is a block diagramillustrating functional configurations of the image capturing device 1,the communication terminal 3, the image management system 5, and thecommunication terminal 7, which constitute a part of the imagecommunication system 10 according the present embodiment. In the imagecommunication system 10 illustrated in FIG. 13, the image managementsystem 5 performs data communication with the communication terminal 3and communication terminal 7 via the communication network 9.

As illustrated in FIG. 13, the image capturing device 1 includes areception unit 12, an image capturing unit 13, a sound collecting unit14, a connection unit 18, and a data storage/read unit 19. Thesefunctional blocks 12 to 19 are implemented by one or more hardwarecomponents illustrated in FIG. 9, when operating in accordance withinstructions from the CPU 111 executing according to the program for theimage capturing device 1, loaded to the DRAM 114 from the SRAM 113.

The image capturing device 1 further includes a memory 1000, which isimplemented by the ROM 112, the SRAM 113, or the DRAM 114 illustrated inFIG. 9.

Hereinafter, a detailed description is given of these functional blocks12 to 19 of the image capturing device 1 with reference to FIGS. 9 and13.

The reception unit 12 of the image capturing device 1 is implemented bythe operation unit 115 and the CPU 111 illustrated in FIG. 9, whichoperate in cooperation with each other, to receive an instruction inputfrom the operation unit 115 according to a user (an installer X in FIG.8) operation.

The image capturing unit 13 is implemented by the imaging unit 101, theimage processor 104, the imaging controller 105, and the CPU 111illustrated in FIG. 9, which operate in cooperation with each other, tocapture an image of the surroundings and thereby acquire thecaptured-image data.

The sound collecting unit 14 is implement by the microphone 108 and thesound processor 109 illustrated in FIG. 9, when operating under controlof the CPU 111, to collect sounds around the image capturing device 1.

The connection unit 18 is implement by an electrical contact, whenoperating under control of the CPU 111. The connection unit 18 isprovided with power from the communication terminal 3, and performs datacommunication.

The data storage/read unit 19 is implement by the CPU 111, whenexecuting according to the program loaded to the DRAM 114, to store dataor information in the memory 1000 and read out data or information fromthe memory 1000.

As illustrated in FIG. 13, the communication terminal 3 includes a dataexchange unit 31, a connection unit 38, and a data storage/read unit 39.These functional blocks 31, 38 and 39 are implemented by one or morehardware components illustrated in FIG. 10, when operating in accordancewith instructions from the CPU 301 executing according to the programsfor the communication terminal 3, loaded to the RAM 303 from the EEPROM304.

The communication terminal 3 further includes a memory 3000, which isimplemented by the ROM 302, the RAM 303, and the EEPROM 304 illustratedin FIG. 10.

Hereinafter, a detailed description is given of these functional blocks31, 38, and 39 with reference to FIGS. 10 and 13.

The data exchange unit 31 of the communication terminal 3 is implementedby the communication unit 313 illustrated in FIG. 10, when operatingunder control of the CPU 301, to exchange data with the image managementsystem 5 via the wireless router 9 a and the communication network 9.

The connection unit 38 is implement by an electrical contact, whenoperating under control of the CPU 301, to supply power to thecommunication terminal 3 and perform data communication.

The data storage/read unit 39 is implement by the CPU 301 illustrated inFIG. 10, when executing according to the program loaded to the RAM 303,to store data or information in the memory 3000 and read out data orinformation from the memory 3000.

It should be noted that a recording medium such as a CD-ROM storing eachof the above-described programs and the HD 504 storing those programsmay be distributed domestically or overseas as a program product.

Hereinafter, a description is given of a functional configuration of theimage management system 5 with reference to FIGS. 11 and 13. The imagemanagement system 5 includes a data exchange unit 51, a detection unit52, and a data storage/read unit 59. These functional blocks 51, 52 and59 are implemented by one or more hardware components illustrated inFIG. 11, when operating in accordance with instructions from the CPU 501executing according to the programs for the image management system 5,loaded to the RAM 503 from the HD 504.

The image management system 5 further includes a memory 5000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11. Thememory 5000 stores the captured-image data transmitted from thecommunication terminal 3. The memory 5000 further stores the score tableillustrated in FIG. 16A and the drawing point table illustrated in FIG.16B. A description is given later of the drawing point table and thedrawing point table.

Further, the memory 5000 includes a person image management DB 5001. Aperson image management table, which is described below, constitutes theperson image management DB 5001. Hereinafter, a detailed description isgiven of the person image management table, the score table, and thedrawing point table.

FIG. 14 is an example of the person image management table. The personimage management table stores a file name of captured image data, acapturing date and time, a person detection area ID, and the positionand area of a person in association with each captured-image ID.

The captured image ID is an example of captured-image identificationinformation for identifying captured image data. The file name ofcaptured image data indicates a file name of the captured image dataidentified by the associated captured-image ID. The capturing date andtime of captured image indicates a date and time at which the associatedcaptured image data is captured by the image capturing device 1identified by a device ID. The captured-image data are stored in thememory 5000.

The person detection area ID is an example of person detection areaidentification information for identifying a person detection area (anarea where a person is detected) detected from the captured-image databy the detection unit 52. The person detection area is represented bythe position and area of a person, as illustrated in FIG. 15. Morespecifically, the person detection area is constituted by a personposition indicating a reference position (x, y) of a person image (arectangular image) and an area indicating a width (w) and a height (h)with reference to this person position. In addition to detecting that acertain person image is included in the captured-image data, thedetection unit 52 distinguishes different persons based on a featureamount such as a height and width size of a person and color. Theposition and area of person indicates a specific point of the personimage in the captured image data (full spherical panoramic image data),and an area specified by a width and a height. For example, asillustrated in FIG. 15, the person image is detected in the form of arectangle. An upper-left corner (x11, y11) of the rectangle indicatesthe reference position of person. The area of person is indicated by thewidth (w11) of the rectangle and the height (h11) of the rectangle.

The captured image ID, the person detection area ID, and the positionand area of person, from among items (field) of the person imagemanagement table, constitute the person detection informationtransmitted at S5 of FIG. 12.

FIG. 16A is an illustration for explaining the score table. The imagemanagement system 5 transmits the score table that is stored in advanceto the communication terminal 7. The communication terminal 7 uses thescore table received from the image management system 5 to generate theheat map image. As illustrated in FIG. 16A, the score table associatestypes of center points with scores that are determined in advance. Thedrawing point is calculated based on the type of the center point fromamong these various types of center points and the associated score. Theimage management system 5 transmits the drawing point table that isstored in advance to the communication terminal 7. The communicationterminal 7 uses the score table received from the image managementsystem 5 to generate the heat map image.

The types of center point includes a center point (basic center point),an average center point (first average center point), and another typeof an average center point (second average center point). The firstaverage center point is an average center point of a plurality of thebasic center points. The second average center point is an averagecenter point of a plurality of the first average center points. Thehighest score is assigned to the second average center point. The secondhighest score is assigned to the first average center point. The centerpoint (basic center point) is assigned with the lowest score.

FIG. 16B is an illustration for explaining the drawing point table. Thedrawing point table stores information of each drawing point associatinga position of each drawing point with a score. The image managementsystem 5 stores in advance the drawing point table containing blank datain the cells for the position of drawing point and the score. The imagemanagement system 5 transmits this drawing point table stored in advanceto the communication terminal 7. The communication terminal 7 stores theposition of the drawing point and the score in the drawing point table,and uses the drawing point table to generate the heat map image.

Hereinafter, a detailed description is given of the functional blocks51, 52 and 59 of the image management system 5 with reference to FIG.13.

The data exchange unit 51 of the image management system 5 isimplemented by the network I/F 509 illustrated in FIG. 11, whenoperating under control of the CPU 501. The data exchange unit 51exchanges data or information with the communication terminal 3 or thecommunication terminal 7 via the communication network 9.

The detection unit 52 is implement by the CPU 501 illustrated in FIG.11, when executing according to the program loaded to the RAM 503. Thedetection unit 52 detects the person image in the captured-image data.The person image is detected, for example, using a person detectionmodel such as the support vector machine (SVM). More specifically, thedetection unit 52 detects each person based on a feature amount of theperson image in the captured-image data. Further, the detection unit 52specifies a position of each detected person image in the captured imagedata (two-dimensional image data). This feature amount is, for example,a height and width size, color, and a face of a person.

The data storage/read unit 59 is implement by the HDD 505, whenoperating under control of the CPU 501, to store data or information inthe memory 5000 and read out data or information from the memory 5000.

Hereinafter, a description is given of a functional configuration of thecommunication terminal 7 with reference to FIGS. 11 and 13. Thecommunication terminal 7 includes a data exchange unit 71, a receptionunit 72, a display control unit 73, a heat map generation unit 74, and adata storage/read unit 79. These functional blocks 71 to 74, and 79 areimplemented by one or more hardware components illustrated in FIG. 11,when operating in accordance with instructions from the CPU 501executing according to the programs for the communication terminal 7,loaded onto the RAM 503 from the HD 504.

The communication terminal 7 further includes a memory 7000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11.

Hereinafter, a detailed description is given of these functional blocks71 to 74 and 79 with reference to FIG. 13.

The data exchange unit 71 of the communication terminal 7 is implementedby the network I/F 509 illustrated in FIG. 11, when operating undercontrol of the CPU 501. The data exchange unit 71 exchanges data orinformation with image management system 5 via the communication network9.

The reception unit 72 is implement by the keyboard 511 and the mouse 512illustrated in FIG. 11, when operating under control of the CPU 501, toreceive an instruction from a user, e.g., the viewer in FIG. 8.

The display control unit 73 is implemented by the CPU 501 illustrated inFIG. 11, when executing according to the program loaded onto the RAM503, to control the display 508 of the communication terminal 7 todisplay images. For example, the display control unit 73 maps the heatmap image on an image (the predetermined-area image, a specific-areaimage, or the captured image) corresponding to the captured-image data.Further, the display control unit 73 controls the display 508 of thecommunication terminal 7 to display the image on which the heat mapimage is mapped.

The heat map generation unit 74 is implement by the CPU 501 illustratedin FIG. 11, when executing according to the program loaded onto the RAM503, to generate the heat map image.

The data storage/read unit 79 is implement by the HDD 505, whenoperating under control of the CPU 501, to store data or information inthe memory 7000 and read out data or information from the memory 7000.

Hereinafter, a detailed description is given of the heat map generationunit 74. The heat map generation unit 74 includes a read-out unit 74 b,a determination unit 74 c, a storage unit 74 d, a calculation unit 74 e,an integration unit 74 f, and a rendering unit 74 h.

The read-out unit 74 b reads out various types of data including thescore table and the drawing point table from the memory 7000, forexample.

The determination unit 74 c determines, based on the person detectioninformation transmitted from the image management system 5, whetherthere is the person detection information in an arbitrary captured-imagedata item.

The storage unit 74 d stores the drawing point, etc., in the memory7000, for example.

The calculation unit 74 e calculates a center point of each persondetection area in a plurality of items of the captured-image data, whichare obtained by continuously capturing objects surrounding the imagecapturing device 1 placed at a specific position. Further, thecalculation unit 74 e calculates a score of the center point based onthe score table to calculate the drawing point. Furthermore, thecalculation unit 74 e specifies the person detection area based on aperson position indicating a position of an arbitrary corner of arectangle area containing a person and an area indicating a width and aheight of the rectangle area with reference to the person position.

In a case in which a plurality of center points in the plurality ofitems of the captured-image data are present in a specific range, theintegration unit 74 f integrates the plurality of center points in thespecific range into a single center point. The integration unit 74 fdetermines a sum of the scores of the plurality of center points as ascore of the integrated center point.

The rendering unit 74 h renders the heat map image using the drawingpoint determined based on the center point calculated by the calculationunit 74 e for weighting of a gradation of color. Specifically, therendering unit 74 h determines the weighting of gradation of color suchthat the higher the score of the center point, the darker the centerpoint. Furthermore, the rendering unit 74 h performs weighting of thegradation of color using the single center point and the sum of scores,which are determined by the integration unit 74 f.

Hereinafter, a description is given of an operation or processing stepsaccording to the present embodiment with reference to FIGS. 17 to 29.

First, the read-out unit 74 b reads out a resolution of the originalcaptured-image data from the memory 7000 (S11). Further, the read-outunit 74 b reads out the person detection information from the memory7000 (S12). Then, the read-out unit 74 b reads out an arbitrarycaptured-image data item that has not yet been acquired from among thegroup of original captured-image data items stored in the memory 7000(S13).

Then, the determination unit 74 c determines whether the persondetection information that is read out at S12 includes the persondetection area ID associated with the same captured image ID as the thatof the newly read-out captured-image data to determine whether a persondetection area is present in the arbitrary captured-image data item thatis read-out at S13 (S14). When the determination unit 74 c determinesthat no person detection area is present in the arbitrary captured-imagedata item that is read-out at S13 (S14: NO), the operation proceeds toS26. By contrast, the determination unit 74 c determines that a persondetection area is present (S14: YES), the calculation unit 74 ecalculates a center point of the person detection area based on theposition and area of person from among the person detection information(S15). Further, at S15, the calculation unit 74 e calculates a score ofa center point p1 using the score table (see FIG. 16A). Specifically, asillustrated in FIG. 19A, when there are four person detection areas inthe arbitrary captured image, for example, the calculation unit 74 efirst selects a person detection area a1, and calculates a center pointp1 of the person detection area a1 as illustrated in FIG. 19B.Subsequently, as illustrated in FIG. 19C, the calculation unit 74 ecalculates a score of the center point p1 using the score table (seeFIG. 16A) to calculate a drawing point s1. This drawing point s1 hasinformation indicating both the position of the center point p1 and thescore of the center point p1. The drawing point is used for weighting ofthe gradation of color when generating the heat map image. Referringagain to FIG. 17, the storage unit 74 d stores the drawing pointinformation indicating the drawing point s1 in the drawing point tablestored in the memory 7000 (S16).

Subsequently, as illustrated in FIG. 18, the determination unit 74 cdetermines whether there is another person detection area that has notyet been selected in the same arbitrary captured-image data item (S17).When the determination unit 74 c determines that there is another persondetection area that has not yet been selected (S17: YES), the sameprocesses as those of S15 and S16 are performed. In other words, thecalculation unit 74 e calculates a center point of a next detection areabased on the position and area of person from among the person detectioninformation (S18). Further, at S18, the determination unit 74 ccalculates a score of a center point p2 using the score table (see FIG.16A). Specifically, as illustrated in FIG. 20A, the calculation unit 74e selects the next person detection area a2 in the same arbitrarycaptured image, and calculates a center point p2 of the person detectionarea a2 as illustrated in FIG. 20B. Then, as illustrated in FIG. 20C,the calculation unit 74 e calculates a score of the center point p2using the score table (see FIG. 16A) to calculate a drawing point s2.This drawing point s2 has information indicating both the position ofthe center point p2 and the score of the center point p2. Referringagain to FIG. 18, the storage unit 74 d stores drawing point informationindicating the drawing point s2 in the drawing point table stored in thememory 7000 (S19).

Subsequently, the determination unit 74 c determines whether one persondetection area contains a center point of another person detection area(S20). When the determination unit 74 c determines that one persondetection area contains no center point of another person detection area(S20: NO), the operation returns to S17. By contrast, when thedetermination unit 74 c determines that one person detection areacontains a center point of another person detection area (S20: YES), thecalculation unit 74 e calculates an average center point using the scoretable (see FIG. 16A) (S21). Specifically, as illustrated in FIG. 21A,when the person detection area a2 contains the center point of theperson detection area a1, the calculation unit 74 e calculates a middleposition between the drawing point s1 and the drawing point s2 as anaverage center point p12. Then, the calculation unit 74 e calculates ascore of the average center point p12 using the score table (see FIG.16A) to calculate a drawing point s12. This drawing point s12 hasinformation indicating both the position of the average center point p12and the score of the average center point p12. Referring again to FIG.18, the storage unit 74 d stores drawing point information indicatingthe drawing point s12 in the drawing point table stored in the memory7000 (S22).

Subsequently, the determination unit 74 c determines whether the averagecenter point calculated by the calculation unit 74 e includes two ormore average center points (S23). Because the calculation unit 74 ecalculates only one average center point s12 so far (S23: NO), theoperation returns to S17. When the determination unit 74 c determinesthat there is a third person detection area a3 as illustrated in FIGS.22A to 22C (S17: YES), the same processes as those of S18 and S19 areperformed on the person detection area a3. Specifically, as illustratedin FIG. 22A, the calculation unit 74 e selects the third persondetection area a3 in the same arbitrary captured image, and calculates acenter point p3 of the person detection area a3 as illustrated in FIG.22B. Then, as illustrated in FIG. 22C, the calculation unit 74 ecalculates a score of the center point p3 using the score table (seeFIG. 16A) to calculate a drawing point s3. This drawing point s3 hasinformation indicating both the position of the center point p3 and thescore of the center point p3.

Further, after S20, the processes of S21 and S22 are performed.Specifically, as illustrated in FIG. 23A, when the person detection areaa3 contains the center point of the person detection area a1 and thecenter point of the person detection area a2, the calculation unit 74 ecalculates a middle position between the drawing point s1 and thedrawing point s3 as an average center point p13 and further calculates amiddle position between the drawing point s2 and the drawing point s3 asan average center point p23. Then, the calculation unit 74 e calculatesa score of the average center point p13 and a score of the averagecenter point p 23 using the score table (see FIG. 16A) to calculate adrawing point s13 and a drawing point s23. The drawing point s13 hasinformation indicating both the position of the average center point p13and the score of the average center point p13. The drawing point s23 hasinformation indicating both the position of the average center point p23and the score of the average center point p23. Then, the storage unit 74d stores drawing point information items respectively indicating thedrawing point s13 and the drawing point s23 in the drawing point tablestored in the memory 7000.

Next, the determination unit 74 c determines whether the average centerpoint calculated by the calculation unit 74 e includes two or moreaverage center points (S23). When the determination unit 74 c determinesthat the average center point calculated by the calculation unit 74 einclude two or more average center points (S23: YES), the calculationunit 74 e calculates an average center point (the second average centerpoint) of the two or more average center points (S24). Further, at S24,the calculation unit 74 e calculates a score of the second averagecenter point using the score table (see FIG. 16A). Specifically, asillustrated in FIG. 24A, the calculation unit 74 e calculates a middleposition (barycenter) of the three drawing points s12, s13, and s23 as asecond average center point p121323. Then, the calculation unit 74 ecalculates a score of the second average center point p121323 using thescore table (see FIG. 16A) to calculate a drawing point s121323. Thisdrawing point s121323 has information indicating both the position ofthe second average center point p121323 and the score of the secondaverage center point p121323. Then, the storage unit 74 d stores drawingpoint information indicating the drawing point s121323 in the drawingpoint table stored in the memory 7000 (S25).

Next, the operation returns to S17 and proceeds to YES, and theprocesses of S18 and S19 are performed. Specifically, as illustrated inFIG. 25A, the calculation unit 74 e selects a fourth person detectionarea a4 in the same arbitrary captured image, and calculates a centerpoint p4 of the person detection area a4 as illustrated in FIG. 25B.Then, as illustrated in FIG. 25C, the calculation unit 74 e calculates ascore of the center point p4 using the score table (see FIG. 16A) tocalculate a drawing point s4. This drawing point s4 has informationindicating both the position of the center point p4 and the score of thecenter point p4. Then, the storage unit 74 d stores drawing pointinformation indicating the drawing point s4 in the drawing point tablestored in the memory 7000. With the processing steps as described sofar, the calculation of the drawing point is performed on all of theperson detection areas in the arbitrary captured-image data item.Accordingly, the operation returns to S17, and the determination unit 74c determines that there is no person detection area that has not yetbeen selected (S17: NO). Thus, the operation proceeds to S26 illustratedin FIG. 17.

Next, the determination unit 74 c determines whether there is acaptured-image data item that has not yet been read out by the read-outunit 74 b in the group of captured-image data items (S26). When thedetermination unit 74 c determines that there is a captured-image dataitem that has not yet been read out (S26: YES), the operation returns toS13.

By contrast, when the determination unit 74 c determines that there isno captured-image data item that has not yet been read out (S26: NO),the integration unit 74 f integrates the drawing points in a specificrange (S27). Further, at S27, the storage unit 74 d stores drawing pointinformation indicating the integrated drawing point in the drawing pointtable. Hereinafter, as illustrated in FIGS. 26A to 26C, a description isgiven of an example in which captured-image data items d1 to d3 that areobtained by capturing an image continuously, respectively containdrawing points s41 to s43 whose positions are different from oneanother. When the three captured-image data items d1 to d3 are overlaidtogether, the drawing point s42 and the drawing point s43 exist in aspecific range (area indicated by a dotted line) with a center of thespecific range at the drawing point s41. In this case, the integrationunit 74 f integrates the three drawing points s41 to s43 into a singledrawing point s414243. The score of the drawing point s414243 is the sumof scores of the three drawing points s41 to s43. It should be notedthat, in a case in which the drawing information indicating theintegrated drawing point s414243 is stored in the drawing point table,the drawing point information items respectively indicating the drawingpoints s41 to s43 are kept stored in the drawing point table. However,instead of storing the drawing point information indicating theintegrated drawing point s414243, the drawing point information itemsrespectively indicating the drawing points s41 to s43 may be deleted.

With the processing steps as described heretofore, the calculation ofthe drawing point is performed on all of the captured-image data items,and all of the drawing point information items are stored in the drawingpoint table. Then, the rendering unit 74 h renders the heat map imagebased on all of the drawing point information items stored in thedrawing point table. FIG. 28 is a view illustrating thepredetermined-area image (including the specific-area image) to whichthe heat map image generated based on the drawing points is attached.The display control unit 73 maps the heat map image on the specific-areaimage and displays the specific-area image on which the heat map imageis mapped on the display 508 of the communication terminal 7. In fact,the heat map is expressed depending on the number of the persondetection areas existing in the plurality of captured-image data itemssuch that red, orange, yellow, yellow-green, and blue spots are renderedin a descending order of the number of the person detection areas.However, in the example illustrated in FIG. 28, the heat map image isexpressed by the gradation of gray color, such that the larger thenumber of the person detection areas, the darker the color.

FIG. 29 is a view illustrating the full spherical panoramic image towhich the heat map image generated based on the drawing points isattached. A illustrated in FIG. 29, in the full spherical panoramicimage, when the virtual camera IC illustrated in FIG. 5 is moved awayfrom the solid sphere CS (when the angle of view is widened), thedisplay control unit 73 controls the heat map image to be placed on afront side of the full spherical panoramic image such that the backsurface of the heat map image is displayed.

Hereinafter, a detailed description is given of display of the heat mapimage illustrated in FIG. 29. FIG. 30 is an illustration for describinga relation between the full spherical panoramic image and the heat mapimage. FIG. 30 illustrates the same view as a top view illustrated inFIG. 5, in which the full spherical panoramic image indicated by adotted line is attached to the solid sphere CS, and heat map images h1and h2 are mapped thereon. When the full spherical panoramic image isviewed from the virtual camera IC, the predetermined-area image of apredetermined area T as a part of the captured-image, a front surface ofthe heat map image h1, and a back surface of the heat map image h2 areviewed. In this case, the heat map image h2 is placed on a front side (aside that is closer to the virtual camera IC) such that the back side ofthe heat map image h2 is displayed.

As described heretofore, according to the first embodiment, therendering unit 74 h uses the drawing point s1 obtained based on thecenter point p1 of the person detection area a1, etc., for the weightingof gradation of color to render the heat map image. The display controlunit 73 maps the heat map image on a captured image corresponding thecaptured-image data, and displays the captured image on which the heatmap image is mapped as illustrated in FIGS. 28 and 29. Thus, the facesor feet of persons are displayed even when the display control unit 73maps the heat map image. Accordingly, a viewer can obtain informationfrom the heat map image and easily recognize a situation represented bythe captured image without being disturbed by the heat map image.

In addition, according to the present embodiment, when displaying thefull spherical panoramic image, the display control unit 73 controls theheat map image to be placed on the front side of the full sphericalpanoramic image, such that the back surface of the heat map image isdisplayed, as illustrated in FIG. 29. Accordingly, the display controlunit 73 is able to display the heat map image three-dimensionally. Thisenables a viewer to recognize easily a situation represented by acaptured image with a heat map displaying method specific to thefull-spherical panoramic image.

Second Embodiment

A second embodiment of the present invention will be describedhereinafter with reference to drawings. It should be noted that adescription is given of just an operation or processing steps becausethe second embodiment is the same or substantially the same as the firstembodiment except for the operation or the processing steps.

Hereinafter, a description is given of an operation or processing stepsaccording to the second embodiment with reference to FIGS. 31 to 34.FIG. 31 is a flowchart illustrating an operation of generating the heatmap image according the second embodiment. FIG. 31 is a flowchartillustrating an operation of generating the heat map image according thesecond embodiment. FIGS. 33A and 33B are illustrations for describing anoperation of determining each of the drawing points (the first to fourthcenter points of the first to fourth person detection areas and thescore of each of the first to fourth center points). FIGS. 34A and 34Bare illustrations for describing an operation of determining a drawingpoint (an average center point between the first center point and thesecond center point and the score thereof), a drawing point (an averagecenter point between the first center point and the third center pointand the score thereof), and an drawing point (an average center point ofthe second center point and the third center point and the scorethereof).

In the first embodiment, the calculation unit 74 e calculates thedrawing points s1 and s2 in the person detection areas a1 and a2 asillustrated in FIGS. 19A to 19C and 20A to 20C, and thereaftercalculates the drawing point s12 based on the average center point p12as illustrated in FIGS. 21A and 21B. Then, the calculation unit 74 ecalculates the drawing points s3 in the person detection areas a3 asillustrated in FIGS. 22A to 22C, and thereafter calculates the drawingpoints s13 and s23 based on the average center points p13 and p23 asillustrated in FIGS. 23A and 23B.

By contrast, in the second embodiment, the calculation unit 74 ecalculates the center points p1 to p3 of the person detection areas a1to a3 together at once as illustrated in FIG. 33A to calculate thedrawing points s1 to s3 together at once as illustrated in FIG. 33B.Then, the calculation unit 74 e calculates the average center pointsp12, p13, and p23 together at once as illustrated in FIG. 34A. Then, thecalculation unit 74 e calculates the drawing points s12, s13, and s23together at once based on the average center points p12, p13, and p23 asillustrated in FIG. 34B. In other words, the processing steps of S111 toS125 in FIGS. 31 and 32 respectively correspond to S11, S12, S13, S14,S15, S16, S21, S22, S23, S24, S25, S16, S26, S27, and S28 in FIGS. 17and 18.

As described heretofore, according to the second embodiment, the sameeffect as the first embodiment is produced.

Third Embodiment

Hereinafter, a description is given of a third embodiment of the presentinvention. An operation of generating a full spherical panoramic image,an overview of a configuration of the image communication system 10, thehardware configurations of the image capturing device 1, thecommunication terminals 3 and 7, and the image management system 5according to the third embodiment are the same as those of the firstembodiment. Therefore, redundant descriptions thereof are omitted below.Hereinafter, a description is given of an overview of processingperformed by the image communication system 10 according to the thirdembodiment with reference to FIG. 35. FIG. 35 is a schematic diagramillustrating operation performed by the image communication system 10according to the third embodiment.

First, the communication terminal 3 acquires the captured-image data,the predetermined-area information, and the related information from theimage capturing device 1 (S101). The communication terminal 3 transmitsthe captured-image data, the predetermined-area image, and the relatedinformation to the image management system 5 (S102). For example, theimage capturing by the image capturing device 1 and the datatransmission from the communication terminal 3 are each performed atcertain time intervals such as every one minute. Then, in response to arequest from the communication terminal 7, the image management system 5transmits the captured-image data, the predetermined-area information,and the related information (S103). Thus, the communication terminal 7displays a predetermined-area image indicated by the predetermined-areainformation in a captured-image corresponding to the captured-image dataas illustrated in FIG. 46A.

Next, in response to a request from the communication terminal 7, theimage management system 5 calculates a position of a person in each of aplurality of captured-image data items that are successively transmittedfrom the communication terminal 3 in chronological order (S104). Then,the image management system 5 transmits, to the communication terminal7, a plurality of captured image IDs and the plurality of foot positioninformation items relating to the plurality of captured-image data itemsobtained by capturing objects at certain time intervals for a certaintime period (S105). The communication terminal 7 generates a heat mapimage based on the plurality of foot position information items.Further, the communication terminal 7 maps the heat map image on apredetermined-area image corresponding to the captured-image dataidentified by any one of the plurality of captured image IDs to displayan image as illustrated in FIG. 46B.

Hereinafter, a description is given of a functional configuration of theimage communication system 10 according to the third embodiment withreference to FIGS. 9 to 11 and 36. FIG. 36 is a block diagramillustrating functional configurations of the image capturing device 1,the communication terminal 3, the image management system 5, and thecommunication terminal 7, which constitute a part of the imagecommunication system 10 according the third embodiment. In the imagecommunication system 10 illustrated in FIG. 36, the image managementsystem 5 performs data communication with the communication terminal 3and communication terminal 7 via the communication network 9.

The functional configurations of the image capturing device 1 and thecommunication terminal 3 are the same or substantially the same as thoseof the first embodiment described with reference to FIG. 13. Therefore,redundant descriptions thereof are omitted below.

Hereinafter, a description is given of a functional configuration of theimage management system 5 with reference to FIGS. 11 and 36. The imagemanagement system 5 includes the data exchange unit 51, the detectionunit 52, a head detection unit 53, a determination unit 54, acalculation unit 55 and the data storage/read unit 59. These functionalblocks 51 to 55 and 59 are implemented by one or more hardwarecomponents illustrated in FIG. 11, when operating in accordance withinstructions from the CPU 501 executing according to the programs forthe image management system 5, loaded to the RAM 503 from the HD 504.The image management system 5 further includes the memory 5000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11. Thememory 5000 stores the captured-image data transmitted from thecommunication terminal 3. Further, the memory 5000 includes the personimage management DB 5001. A person image management table, which isdescribed below, constitutes the person image management DB 5001.Hereinafter, a detailed description is given of the person imagemanagement table.

FIG. 37 is an example of the person image management table according tothe third embodiment. The person image management table stores a filename of captured image data, a capturing date and time, a face ID, andfoot position information in association with each captured-image ID.

The captured image ID is an example of captured-image identificationinformation for identifying captured image data. The file name ofcaptured image data indicates a file name of the captured image dataidentified by the associated captured-image ID. The capturing date andtime of captured image indicates a date and time at which the associatedcaptured image data is captured by the image capturing device 1identified by a device ID. The captured-image data are stored in thememory 5000.

The face ID is an example of face identification information foridentifying a person's head including a face detected from thecaptured-image data by the head detection unit 53. The foot positioninformation indicates a position of foot in the captured-image datacalculated by the calculation unit 55. In this case, the position offoot indicates a two-dimensional position of foot in the Mercator imageas the captured image (see FIG. 3C). It should be noted that theposition of foot may be indicated by (x, y)=(latitude, longitude).

Hereinafter, a detailed description is given of each of the functionalblocks 51 to 55 and 59 of the image management system 5 with referenceto FIG. 36.

It should be noted that functions implemented by the data exchange unit51 and the data storage/read unit 59 are the same or substantially thesame as those of the first embodiment. Therefore, redundant descriptionsthereof are omitted below.

The detection unit 52 is implement by the CPU 501 illustrated in FIG.11, when executing according to the program loaded onto the RAM 503. Thedetection unit 52 according to the third embodiment detects a zenithdirection based on the orientation and tilt information of the relatedinformation received at S2.

The head detection unit 53 is implement by the CPU 501 illustrated inFIG. 11, when executing according to the program loaded to the RAM 503.The head detection unit 53 detects an arbitrary person's head includinga face in the captured-image data. This detection of head is performed,for example, using a person detection model such as the support vectormachine (SVM). More specifically, the head detection unit 53 detectseach person's head including a face based on a feature amount of atwo-dimensional face image in the captured-image data. Further, the headdetection unit 53 specifies a position of the detected head in thecaptured image data. For example, as illustrated in FIG. 38, the imageof the head including face is detected in the form of a rectangle. Acenter (x11, y11) of the rectangle indicates the position of theperson's head. In addition, in FIG. 38, the position of foot isindicated by coordinates (x21, y21). The coordinates are stored as thefoot position information in the person image management tableillustrated in FIG. 37.

The determination unit 54 is implement by the CPU 501 illustrated inFIG. 11, when executing according to the program loaded to the RAM 503.The determination unit 54 determines whether an arbitrary person's headis detected by the head detection unit 53.

The calculation unit 55 is implement by the CPU 501 illustrated in FIG.11, when executing according to the program loaded to the RAM 503. Thecalculation unit 55 calculates a size (“2R” described later) of the headdetected by the head detection unit 53. Further, the calculation unit 55calculates a distance (“h” described later) between the center of thehead and the foot based on the calculated head's size and an averageperson's body size that is determined in advance. Furthermore, thecalculation unit 55 calculates a position that is away from the centerposition of the head by the distance (h) between the center of the headand the foot as the position of foot in the captured-image data. Adetailed description is given later of this operation performed by thecalculation unit 55.

Hereinafter, a description is given of a functional configuration of thecommunication terminal 7 with reference to FIGS. 11 and 36. Thecommunication terminal 7 according to the third embodiment includes thedata exchange unit 71, the reception unit 72, the display control unit73, the heat map generation unit 74, and the data storage/read unit 79.These functional blocks 71 to 74, and 79 are implemented by one or morehardware components illustrated in FIG. 11, when operating in accordancewith instructions from the CPU 501 executing according to the programsfor the communication terminal 7, loaded onto the RAM 503 from the HD504.

The communication terminal 7 further includes the memory 7000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11.

Hereinafter, a detailed description is given of these functional blocks71 to 74 and 79 with reference to FIG. 36.

It should be noted that functions implemented by the data exchange unit71, the reception unit 72, the data storage/read unit 79 are the same orsubstantially the same as those of the first embodiment. Therefore,redundant descriptions thereof are omitted below.

The display control unit 73 is implemented by the CPU 501 illustrated inFIG. 11, when executing according to the program loaded onto the RAM503, to control the display 508 of the communication terminal 7 todisplay images. For example, the display control unit 73 maps the heatmap image on an image (the predetermined-area image or the capturedimage) corresponding to the captured-image data. Further, the displaycontrol unit 73 controls the display 508 of the communication terminal 7to display the image on which the heat map image is mapped.

The heat map generation unit 74 is implement by the CPU 501 illustratedin FIG. 11, when executing according to the program loaded onto the RAM503. The heat map generation unit 74 according the third embodimentgenerates the heat map image based on the foot position information.Specifically, the heat map generation unit 74 uses each of the footpositions in the plurality of time-series captured-image data items thatare obtained by continuously capturing images by the image capturingdevice 1 at certain time intervals for weighting of a gradation of colorto render the heat map image.

Hereinafter, a description is given of an operation or processing stepsaccording to the third embodiment with reference to FIG. 39 to FIGS. 47Aand 47B. FIG. 39 is a flowchart illustrating an operation of specifyinga position of a person's foot.

First, the data storage/read unit 59 reads out, from the memory 5000,the captured-image data in which a person's face is to be detected(S211). The detection unit 52 detects a zenith direction based on therelated information transmitted from the communication terminal 3 atS102 (S212). The head detection unit 53 detects an arbitrary person'shead including a face in the captured-image data (S213). Thedetermination unit 54 determines whether an arbitrary person's head isdetected by the head detection unit 53 (S214). When the determinationunit 54 determines that the arbitrary person's head is detected (S214:YES), the calculation unit 55 calculates a size (“2R” described later)of the detected head (S215).

Next, the calculation unit 55 calculates a distance (“h” describedlater) between the center of the head and the foot based on thecalculated size of head and an average person's body size that isdetermined in advance (S216). Furthermore, the calculation unit 55calculates a position that is away from the center position of the headby the distance (h) between the center of the head and the foot as aposition of foot in the captured-image data (S217).

Next, the data storage/read unit 59 stores, in the person imagemanagement table, person image information including the captured imageID, the file name of the captured image data, the capturing date andtime, the face ID, the foot position information (S218).

Next, the operation returns to S213, and the head detection unit 53detects another person's face in the same captured-image data as thecaptured-image data on which the head detection unit 53 first performedthe detection of person's head. Thus, the processing from S213 to S218is repeated until all person's faces are detected. When no person's faceis detected (S214: NO), it is determined that all persons' faces aredetected, and the operation of specifying the position of person's footends.

Hereinafter, a detailed description is given of the processing from S215to S217 with reference to FIGS. 40 to FIGS. 46A and 46B. FIG. 40 is aplan view illustrating an example situation where a full sphericalpanoramic image is captured by the image capturing device 1. FIG. 41 isa side view illustrating an example situation where a full sphericalpanoramic image is captured by the image capturing device 1.

FIGS. 40 and 41 illustrate an example in which there is an obstacle 2001on a floor 2000 between a person 900 and the image capturing device 1,and therefore a space around a foot 904 of the person is not captured byimage capturing device 1. Even in this case, the image management system5 detects a head (face) 901 of the person 900 using a recent high-levelface detection (recognition) technique, and specifies a position of thefoot 904 of the person 900 based on the position of head 901.Accordingly, the position of the foot 904 is specified even when thereis the obstacle 2001. It should be noted that a foot detection(recognition) technique is less accurate than a face detectiontechnique, and therefore the position of the foot 904 is specified usinga face detection (recognition) technique also when there is no obstacle2001.

Hereinafter, a further detailed description is given of specifying theposition of the foot 904. In FIGS. 40 and 41, assuming that the imagingelement 103 a (103 b) of the image capturing device 1 is a center of acircle, an angle “a” represents an angle between a horizontal line drawnfrom the imaging element 103 a (103 b) and a virtual line drawn from theimaging element 103 a (103 b) through a center of the head (face) 901 ofthe person 900. Further, assuming that the imaging element 103 a (103 b)of the image capturing device 1 is the center of a circle, an angle “β”represents an angle between a virtual line drawn from the imagingelement 103 a (103 b) through a top of the head 901 of the person 900and a virtual line drawn from the imaging element 103 a (103 b) througha bottom of the head (face) 901 of the person 900. The image managementsystem 5 according to the third embodiment obtains an angle “θ” betweenthe horizontal line drawn from the imaging element 103 a (103 b) and avirtual line drawn from the imaging element 103 a (103 b) through thefoot 904 of the person 900 to specify the position of the foot 904.

In addition, in a case in which the image capturing device 1 capturesthe person 900 and the obstacle 2001, the full spherical panoramicimage, which is represented as an image facing toward a center of thesphere as illustrated in FIG. 4A, includes a head (face) image 911 as animage of the head (face) 901, an upper extremity image 912 as an imageof an upper portion of an upper extremity 902, and an obstacle image2011 as an image of the obstacle 2001. In other words, in a situation asillustrated in FIGS. 40 and 41, the image capturing device 1 cannotcapture a lower portion of the upper extremity 902 and a lower extremity903 (including the foot 904).

It should be noted that, in this case, the head 901 approximates asphere. The angle β representing a longitudinal length of the head 901can be calculated as an actual measured value using a face detection(recognition) technique. Further, in the present embodiment, it is notthe actual size and length but the relative size and length thatmatters. Accordingly, when specifying the position of the foot 904, thecalculation of the position of foot is calculated assuming that thedistance between the head 901 of the person 900 and the imaging element103 a (103) is 1 (circle unit).

Taking a usual human figure (head:height=1:n, an average of the Japaneseis 7.3) into consideration, the angle θ is expressed by the followingequation (2).

$\begin{matrix}{\theta = {\arctan\left\{ \frac{{\sin(\alpha)} + {\left( {{2n} - 1} \right) \cdot {\sin\left( {\frac{1}{2}\beta} \right)}}}{\cos(\alpha)} \right\}}} & \left( {{Equation}\mspace{14mu} 2} \right)\end{matrix}$

Hereinafter, a description is given of how the equation (2) is obtained.

(1) Calculation of Size of Head 901 (S215, FIG. 42)

Referring to FIG. 42, assuming that a position of the imaging element103 a (103 b) is “o1” (circle unit), a center of the head 901 of theperson 900 is “o2”, a point of contact where a virtual line drawn fromo1 through a top of the head 901 contacts a circumference of a circlewhose center point is at o2 is “A”, a distance R between o2 and A isexpressed by the following equation (3).

$\begin{matrix}{R = {{\sin\left( {\frac{1}{2}\beta} \right)}.}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

Accordingly, 2R, which is a size of the head 901 (a diameter of the head901), is calculated using the following equation (4).

$\begin{matrix}{{2R} = {2{{\sin\left( {\frac{1}{2}\beta} \right)}.}}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

(2) Calculation of the Distance Between the Center of the Head 901 andthe Foot 904 (S216, FIG. 43)

Next, referring to FIG. 43, a person's height H is expressed by thefollowing equation (5).

$\begin{matrix}{H = {2{{\sin\left( {\frac{1}{2}\beta} \right)} \cdot {n.}}}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

Accordingly, a distance h from the center o2 of the head 901 to aposition B of the foot 904 is calculated by subtracting a half of thesize of the head 901 from the height using the following equation (6).

$\begin{matrix}{h = {{{2{{\sin\left( {\frac{1}{2}\beta} \right)} \cdot n}} - {\sin\left( {\frac{1}{2}\beta} \right)}} = {\left( {{2n} - 1} \right) \cdot {{\sin\left( {\frac{1}{2}\beta} \right)}.}}}} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

(3) Specification of the Position of the Foot 904 (S217, FIGS. 44 and45)

Next, referring to FIG. 44, assuming that an intersection point where anextension drawn from the position B of the foot 904 through the centero2 of the head 901 intersects a horizontal line drawn from the positiono1 of the imaging element 103 a (103 b) is “C”, from among three sidesof a right triangle defined by o1, C and B, a length of a side definedby o1 and C is cos(α), and a length of a side defined by C and o2 issin(α). Accordingly, a length of a side defined by C and B is calculatedby adding the length of the side defined by o1 and C to the length ofthe side defined by C and o2 using the following equation (7).

$\begin{matrix}{{CB} = {{\sin(\alpha)} + {\left( {{2n} - 1} \right) \cdot {{\sin\left( {\frac{1}{2}\beta} \right)}.}}}} & \left( {{Equation}\mspace{14mu} 7} \right)\end{matrix}$

Further, referring to FIG. 45, the angle θ between the horizontal linedrawn from the position o1 of the imaging element 103 a (103 b) and avirtual line drawn from the imaging element 103 a (103 b) through thefoot 904 of the person 900 is calculated by arctan (the length of theside defined by C and B/the length of the side defined by o1 and C).Accordingly, the angle θ is calculated using the above equation (2).

With the operation described heretofore, as illustrated at S105 of FIG.35, the foot position information indicating the calculated position ofthe foot 904 is transmitted to the communication terminal 7 togetherwith the captured image ID of the captured-image data from which thefoot has been detected. Then, as illustrated at S106 of FIG. 35, theheat map generation unit 74 of the communication terminal 7 generatesthe heat map image. The display control unit 73 maps the heat map imageon the predetermined-area image, and control the display 508 of thecommunication terminal 7 to display the image on which the heat mapimage is mapped.

FIG. 46A is an example of the predetermined-area image on which the heatmap image is not mapped. FIG. 46B is an example of thepredetermined-area image on which the heat map image is mapped. Asillustrated in FIG. 46B, the heat map image represents a trajectory of aperson's foot. Accordingly, compared with an image on which the heat mapimage is mapped at or near a barycenter of a person as illustrated inFIG. 47B, a user (viewer Y) viewing the image as illustrated in FIG. 46Bcan easily recognize the position of a person.

In the images illustrated in FIGS. 46B and 47B, the heat map is actuallyexpressed depending on the number of the detected heads (feet) such thatred, orange, yellow, yellow-green, and blue spots are rendered in adescending order of the number of the detected heads (feet). However, inthe example illustrated in FIGS. 46B and 47B, the number of the persondetection areas is expressed by the gradation of gray color, such thatthe larger the number of the person detection areas, the darker thecolor.

As described heretofore, according to the third embodiment of thepresent invention, when specifying a person's foot position in thecaptured-image data, not a person's foot but a person's face is detectedusing a face detection technique that is relatively advanced, andthereafter the position of foot is calculated based on the position ofhead including a face. This enables the heat map image to express adepth as illustrated in FIG. 46B. Accordingly, a viewer can easilyrecognize a track of a person based on the heat map image.

Further, in a place such as a store where sales items are placed, a footmay not be projected on the captured image because the foot is hidden byan obstacle. Even in such case, because the position of foot iscalculated based on the detection of face, the heat map image accordingto the third embodiment can effectively express a track of person in astore or the like. Especially in a shoes store, a shoe is likely to bedetected (recognized) as a person's foot in error even if a footdetection (recognition) technique advance. Even in such case, becausethe position of foot is calculated based on the detection of face, theheat map image according to the third embodiment can effectively expressa track of person in a shoe store.

The image management system 5 is implemented by either a single computeror a plurality of computers, each including or performing at least apart of the functional blocks, operations, or memories of the imagemanagement system 5 as described above.

A recording medium such as a CD-ROM storing the programs in the aboveembodiment and the HD 504 storing those programs may be distributeddomestically or internationally as a program product.

Further, in alternative to the laptop computer, the communicationterminal 7 may be implemented by a personal computer such as a desktopcomputer, tablet, a smartphone, a tablet, or a smartwatch.

Fourth Embodiment

Hereinafter, a description is given of a fourth embodiment of thepresent invention. An operation of generating a full spherical panoramicimage, an overview of a configuration of the image communication system10, the hardware configurations of the image capturing device 1, thecommunication terminals 3 and 7, and the image management system 5according to the fourth embodiment are the same as those of the firstembodiment. Therefore, redundant descriptions thereof are omitted below.

Hereinafter, a description is given of an overview of processingperformed by the image communication system 10 according to the fourthembodiment with reference to FIG. 48. FIG. 48 is a schematic diagramillustrating operation performed by the image communication system 10according to the fourth embodiment.

First, the communication terminal 3 acquires the captured-image data,the predetermined-area information, and the related information from theimage capturing device 1 (S301). The communication terminal 3 transmitsthe captured-image data, the predetermined-area image, and the relatedinformation to the image management system 5 (S302). The communicationterminal 3 repeats the processing of S301 and S302. The processing ofS301 and S302 is repeated at irregular intervals in time. For example,the processing is first repeated 5 seconds after a first processing andthereafter again repeated 6 seconds after the second processing.Alternatively, the processing of S301 and S302 may be repeated atregular intervals in time such as every 5 seconds.

Next, the image management system 5 detects an area of a person image,which is an image of a person to generate person detection information(S303). The area of the person image may be referred to as “persondetection area” hereinafter. The image management system 5 transmits thepredetermined-area image and the related information (S304). Next, inresponse to a request from the communication terminal 7, the imagemanagement system 5 transmits the person detection information and agroup of captured-image data items (S305). Thus, the communicationterminal 7 stores, in the memory 7000 (described later), the persondetection information and the group of captured-image data items.

Next, the communication terminal 7 generates a heat map image asillustrated in FIG. 56 (S306). This enables a viewer Y viewing the heatmap image to recognize how each person stays in a certain place withoutviewing a plurality of full spherical panoramic images (orpredetermined-area images) captured for a certain period of time.

Hereinafter, a description is given of a functional configuration of theimage communication system 10 according to the fourth embodiment withreference to FIGS. 9 to 11 and 49. FIG. 49 is a block diagramillustrating functional configurations of the image capturing device 1,the communication terminal 3, the image management system 5, and thecommunication terminal 7, which constitute a part of the imagecommunication system 10 according the fourth embodiment. In the imagecommunication system 10 illustrated in FIG. 49, the image managementsystem 5 performs data communication with the communication terminal 3and communication terminal 7 via the communication network 9.

The functional configurations of the image capturing device 1 and thecommunication terminal 3 according to the fourth embodiment are the sameor substantially the same as those of the first embodiment describedwith reference to FIG. 13. Therefore, redundant descriptions thereof areomitted below.

Hereinafter, a description is given of a functional configuration of theimage management system 5 according to the fourth embodiment withreference to FIGS. 11 and 49. The image management system 5 includes thedata exchange unit 51, the detection unit 52, and the data storage/readunit 59. These functional blocks 51, 52 and 59 are implemented by one ormore hardware components illustrated in FIG. 11, when operating inaccordance with instructions from the CPU 501 executing according to theprograms for the image management system 5, loaded to the RAM 503 fromthe HD 504.

The image management system 5 further includes the memory 5000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11. Thememory 5000 stores the captured-image data transmitted from thecommunication terminal 3.

Further, the memory 5000 includes the person image management DB 5001.The person image management table illustrated in FIG. 14 constitutes theperson image management DB 5001. The person image management tableaccording this fourth embodiment is the same or substantially the sameas that of the first embodiment. Therefore, a redundant descriptionthereof is omitted below.

Further it should be noted that the functions implemented by the dataexchange unit 51, the detection unit 52 and the data storage/read unit59 are the same or substantially the same as those of the firstembodiment. Therefore, redundant descriptions thereof are omitted below.

Hereinafter, a description is given of a functional configuration of thecommunication terminal 7 according to the fourth embodiment withreference to FIGS. 11 and 49. The communication terminal 7 according tothe fourth embodiment includes the data exchange unit 71, the receptionunit 72, the display control unit 73, the heat map generation unit 74,and the data storage/read unit 79. These functional blocks 71 to 74, and79 are implemented by one or more hardware components illustrated inFIG. 11, when operating in accordance with instructions from the CPU 501executing according to the programs for the communication terminal 7,loaded onto the RAM 503 from the HD 504.

The communication terminal 7 further includes the memory 7000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11.

Hereinafter, a detailed description is given of these functional blocks71 to 74 and 79 with reference to FIG. 49.

It should be noted that functions implemented by the data exchange unit71, the reception unit 72, the display control unit 73, and the datastorage/read unit 79 are the same or substantially the same as those ofthe first embodiment. Therefore, redundant descriptions thereof areomitted below.

The heat map generation unit 74 is implement by the CPU 501 illustratedin FIG. 11, when executing according to the program loaded onto the RAM503, to generate the heat map image.

Hereinafter, a detailed description is given of the heat map generationunit 74 according to the fourth embodiment. The heat map generation unit74 includes the read-out unit 74 b, the determination unit 74 c, thestorage unit 74 d, the calculation unit 74 e, an addition unit 74 g, andthe rendering unit 74 h.

The read-out unit 74 b reads out various types of data from the memory7000, for example.

The determination unit 74 c according to the forth embodiment determineswhether a first position and a first size of a first detection areawhere a first person is detected in a first captured-image data itemfrom among the plurality of captured-image data items is within apredetermined range when compared with a second position and a secondsize of a second detection area where a second person is detected in asecond captured-image data item that is captured after the capturing ofthe first captured-image data item. The determination unit 74 c therebydetermines whether the first person and the second person are a sameobject based on the determination result.

The storage unit 74 d stores various types of data in the memory 7000,for example.

The calculation unit 74 e according to the fourth embodiment calculates,for example, a position of a center point of the person detection areaand a size of the person detection area based on the position and areaof person from among the person detection information.

The addition unit 74 g stores a time difference in a column for a staypoint in an intermediate table (described below) as illustrated in FIG.53C.

The rendering unit 74 h renders the heat map image using a final tableillustrated in FIG. 55 for weighting of a gradation of color.

Hereinafter, a description is given of an operation or processing stepsaccording to the fourth embodiment with reference to FIGS. 50 to 54.FIGS. 50 and 51 is a flowchart illustrating an operation of generatingthe heat map image according the fourth embodiment. FIGS. 52A to 52D,53A to 53D, 54A to 54D and 55 are illustrations for explaining how astay point is calculated. Specifically, FIGS. 52A, 53A and 54A areschematic views each illustrating a captured image including the persondetection areas. Further, FIGS. 52B, 53B and 54B are example tables eachindicating person specification information. Furthermore, FIGS. 52C, 53Cand 54C are each an example of the intermediate table. Still further,FIGS. 52D, 53D and 54D and FIG. 55 are each an example of the finaltable.

First, the read-out unit 74 b reads out a resolution of the originalcaptured-image data from the memory 7000 (S311). Further, the read-outunit 74 b reads out all of the person detection information items fromthe memory 7000 (S312). Then, the read-out unit 74 b searches the memory7000 for a captured-image data item that has not yet been read out(S313).

Next, the determination unit 74 c determines whether there is acaptured-image data item that has not yet been read out by the search atS313 in the memory 7000 (S314). When the determination unit 74 cdetermines that there is a captured-image data item that has not yetbeen read out (S314: YES) in the memory 7000, the read-out unit 74 breads out the captured-image data item that has not yet been read outfrom the memory 7000 (S315). By contrast, when the determination unit 74c determines that there is no captured-image data item that has not yetbeen read out (S314: NO), the operation proceeds to S341.

Next, the determination unit 74 c determines whether the persondetection information items that are read out at S312 contain a persondetection information item specified by the same captured image ID asthe captured image ID indicating the newly read-out captured-image dataitem. Based on a result of this determination, the determination unit 74c determines whether a next person detection area is present in the samecaptured-image data item that is read out at S315 (S316). The processingof S316 is processing of searching for the person detection area as acomparison source, which is described below. It should be noted thatwhen the determination at S316 is performed for the first time, thedetermination unit 74 c determines whether there is a “first” persondetection area instead of the next person detection area. Assuming thatall of the person detection information items that are read out at S312are as illustrated in FIG. 14, in the determination that is performedfor the first time, the determination unit 74 c determines that there isa person detection information item indicated by a person detection areaID “a0011” as the person detection information specified by the capturedimage ID “p001”. In the next determination, the determination unit 74 cdetermines that there is the next person detection information itemindicated by a person detection area ID “a0012” as the person detectioninformation specified by the same captured image ID “p001”.

When the determination unit 74 c determines that the next (or first)person detection area is present in the same captured-image data itemthat is read out at S315 (S316: YES), the calculation unit 74 ecalculates the position of center point of the person detection area anda size of the person detection area based on the position and area ofperson from among the person detection information (S317). Specifically,the calculation unit 74 e specifies the person detection area based on aposition of an arbitrary corner of a rectangle area containing a personand an area indicating a width and a height of the rectangle area withreference to the position of corner. It should be noted that whencalculating the size of the person detection area, the calculation unit74 e uses information indicating the resolution that is read out atS311.

For example, as illustrated in FIG. 52A, when four person detectionareas are present in a captured-image data item of a first capturingdate and time, the calculation unit 74 e first calculates a position(x1, y1) of a center point pa1 of a person detection area a1 and a sizeα1 of the person detection area a1 based on information of a person'sarea (x11, y11, w11, h11) of the person detection information identifiedby the person detection area ID “a0011”. Next, the storage unit 74 dassigns an identifier to the center point position and the sizecalculated at S317, and temporarily stores the position of the centerpoint and the size in association with the identifier as personspecification information in the memory 7000 as illustrated in FIG. 52B(S318). In the person specification information illustrated in FIG. 52B,the center point position (x1, y1) and the size α1 are associated withan identifier “a01”. The person specification information illustrated inFIG. 52B includes no similarity determination result. This is because noperson detection area to be compared with the person detection area inthe first captured-image data is present, because there is no previouscaptured-image data item that is captured before the firstcaptured-image data item.

It should be noted that the person detection areas a1, a2, a3, and a4contain the center points pa1, pa2, pa3, and pa4, respectively. Further,the person detection areas a1, a2, a3, and a4 are identified by theidentifiers a01, a02, a03, and a04, respectively.

Subsequently, the operation returns to S316, and the determination unit74 c determines whether there is a next person detection area in thesame captured-image data item that is read out at S315. Assuming thatall of the person detection information items that is read out at S312are as illustrated in FIG. 14, the determination unit 74 c determinesthere is a person detection information item indicated by a persondetection area ID “a0012” as a next person detection information itemspecified by the same captured image ID “p001”. Then, at S317, thecalculation unit 74 e calculates a position (x2, y2) of a center pointpa2 of a person detection area a2 and a size α2 of the person detectionarea a2 based on information of a person's area (x12, y12, w12, h12) ofthe person detection information identified by the person detection areaID “a0012”. Next, at S318, the storage unit 74 d assigns an identifierto the center point position and the size calculated at S317, andtemporarily stores the position of the center point and the size inassociation with the identifier in the memory 7000 as illustrated inFIG. 52B. In FIG. 52B, the center point position (x2, y2) and the sizeα2 are stored in association with an identifier “a02”.

The processing of S316 to S318 as above is repeated until thedetermination unit 74 c determines that there is no next persondetection area in the same captured-image data item that is read out atS315 (S316: NO), and the operation proceeds to S321 of FIG. 51.

Next, the determination unit 74 c determines whether there is a personwho is staying in one place for more than a certain period of time(referred to as a “staying person” hereinafter) or a potential stayingperson (S321). Specifically, when determining whether there is apotential staying person, the determination unit 74 c determines whetherthe identifier that is temporarily stored at S318 has been alreadymanaged in the intermediate table. When the identifier that istemporarily stored at S318 is not managed in the intermediate table, thedetermination unit 74 c determines that there is no potential stayingperson.

Further, when determining whether there is a staying person, thedetermination unit 74 c determines whether the identifier that istemporarily stored as the similarity determination result at S318 ismanaged in the intermediate table. In other words, when determiningwhether there is a staying person, the determination unit 74 cdetermines whether the intermediate table already manages an identifierindicating the person detection area that is the same as or similar tothe person detection area indicated by the identifier that istemporarily stored at S318. When such identifier is not managed in theintermediate table, the determination unit 74 c determines that there isno staying person.

For example, the intermediate table initially manages no information,the determination unit 74 c determines each person corresponding to eachof the all person specification information items illustrated in FIG.52B as a potential staying person.

When the determination unit 74 c determines that there is no stayingperson or potential staying person (S321: NO), the data storage/readunit 79 stores, in the intermediate table stored in the memory 7000,each identifier of the person detection areas as new comparison sourcesin association with a stay point. In the first processing, because thereis no captured-image data item that is captured at a previous capturingdate and time, the identifiers indicating all of the person detectionareas are initially stored in the intermediate table in association withzero stay point as illustrated in FIG. 52C. It should be noted that theintermediate table stores intermediate data associating each identifierof the person detection area with the stay point.

Next, the calculation unit 74 e calculates a time difference between thecapturing date and time of the captured-image data item that iscurrently being processed and the previous capturing date and time ofthe captured-image data (S323). In this example, the time differencebetween the two capturing date and times is zero (second), because nocaptured-image data item is present that is captured at the previouscapturing date and time. The addition unit 74 g adds the time differenceto a stay point column in the intermediate table (S324). Because thecaptured-image data item that is captured at the first capturing dateand time is a current processing target, each stay point associated withall of the identifiers is zero.

Subsequently, the operation proceeds to S314, and the processing isrepeated from S314. It should be noted that, at this point of time, nodata is managed in the final table illustrated in FIG. 18D.

Then, at S314, the determination unit 74 c determines whether the memory7000 stores a captured-image data item that has not yet been read out.When the determination unit 74 c determines that there is acaptured-image data item that has not yet been read out in the memory7000 (S314: YES), the read-out unit 74 b reads out a captured-image dataitem captured at a second capturing date and time that has not yet beenread out from the memory 7000 (S315).

Next, at S316, the determination unit 74 c determines whether there is anext person detection area in the same captured-image data item of thesecond capturing date and time that is read out at S315. It should benoted that when the determination at S316 is performed for the firsttime on the captured-image data of the second capturing date and time,the determination unit 74 c determines whether there is a “first” persondetection area instead of the next person detection area.

Assuming that all of the person detection information items that areread out at S312 are as illustrated in FIG. 14, for the captured-imagedata item captured at the second capturing date and time, thedetermination unit 74 c determines that there is a person detectioninformation item indicated by a person detection area ID “a0021” as afirst person detection information item specified by the captured imageID “p0012” at S316. For example, when four person detection areas arepresent as illustrated FIG. 53A, the calculation unit 74 e calculatesthe position (x1, y1) of a center point pb1 of a person detection areab1 and the size α1 of the person detection area b1 based on informationof a person's area (x21, y21, w21, h21) of the person detectioninformation identified by the person detection area ID “a0021” at S317.Next, at S318, the storage unit 74 d assigns an identifier to the centerpoint position and the size calculated at S317, and temporarily storesthe position of the center point and the size in association with theidentifier as person specification information in the memory 7000 asillustrated in FIG. 53B. In FIG. 53B, the center point position (x1, y1)and the size α1 are stored in association with an identifier “b01”.

It should be noted that the person detection areas b1, b2, b3, and b4contain the center points pb1, pb2, pb3, and pb4, respectively. Further,the person detection areas b1, b2, b3, and b4 are identified by theidentifiers b01, b02, b03, and b04, respectively.

Next, the operation returns to S316, and the determination unit 74 cdetermines whether a next person detection area is present in the samecaptured-image data item (see FIG. 53A) that is read out at S315. Inthis example, assuming that all of the person detection informationitems that are read out at S312 are as illustrated in FIG. 14, in thedetermination that is performed for the second time, the determinationunit 74 c determines that there is a person detection information itemindicated by a person detection area ID “a0022” as the person detectioninformation specified by the same captured image ID “p002”. Then, atS317, the calculation unit 74 e calculates the position (x2, y2) of thecenter point pb2 of the person detection area b2 and a size α2 of theperson detection area b2 based on information of a person's area (x22,y22, w22, h22) of the person detection information identified by theperson detection area ID “a0022” as illustrated in FIG. 53A. Next, atS318, the storage unit 74 d assigns an identifier to the center pointposition and the size calculated at S317, and temporarily stores theposition of the center point and the size in association with theidentifier as person specification information in the memory 7000 asillustrated in FIG. 53B. In FIG. 53B, the center point position (x2, y2)and the size α2 are stored in association with an identifier “b02”.

The processing of S316 to S318 as above is repeated until thedetermination unit 74 c determines that there is no next persondetection area in the same captured-image data item of the secondcapturing date and time that is read out at S315 (S316: NO), and theoperation proceeds to S321 of FIG. 51.

Subsequently, when the determination unit 74 c determines that there isa staying person or a potential staying person (S321: YES), thedetermination unit 74 c determines whether the captured-image data itemof the previous capturing date and time illustrated in FIG. 52B containsa next person detection area that is close to the person detection areaas a comparison source indicated by each of the identifiers in theperson specification information illustrated in FIG. 53B (S325). Itshould be noted that when the determination at S325 is performed for thefirst time, the determination unit 74 c determines whether there is a“first” person detection area close to the person detection area as acomparison source instead of the next person detection area. In thisexample, the determination unit 74 c determines that, in FIG. 52B, thereis the person detection area a1 indicated by the identifier 01 as acomparison destination.

Next, the determination unit 74 c determines a degree of similaritybetween the person detection area as a comparison source in thecaptured-image data item of the currently-processed capturing time andan arbitrary person detection area as a comparison destination in thecaptured-image data item of the previous capturing time (S326). Morespecifically, the determination unit 74 c determines whether a distancebetween the center point of the person detection area as the comparisonsource and the center point of the person detection area as thecomparison destination is within a predetermined value, such as 10pixels. Further, the determination unit 74 c determines whether a ratiobetween the size of the person detection area as the comparison sourceand the size of the person detection area as the comparison destinationis within a predetermined value such as 80% to 120%. When thedetermination unit 74 c determines that the distance and the ratio ofsize are within the predetermined values (S326: YES), the storage unit74 d temporarily stores a similarity determination result as illustratedin FIG. 53B (S327). For example, when the person detection area a1 (seeFIG. 52A) as a first comparison destination is the same as or similar tothe person detection area b1 (see FIG. 53A) as the comparison sourceindicated by the identifier b01, the storage unit 74 d stores theidentifier 01 of the person detection area a1 in association with theidentifier b01 as the similarity determination result.

Next, the calculation unit 74 e calculates a time difference between thecapturing date and time of the captured-image data item that iscurrently being processed and the previous capturing date and time ofthe captured-image data (S328). The addition unit 74 g adds the timedifference calculated at S328 to a stay point associated with theidentifier a01 indicated as the similarity determination result in theintermediate table as illustrated in FIG. 53C (S329). In this example,as illustrated in FIG. 14, a time difference between thecurrently-processed captured-image data item (captured image ID “p001”)and the previous captured-image data item (captured image ID “p001”) isfive seconds. In addition, a value of the stay point associated with theidentifier a01 in a previous intermediate table illustrated in FIG. 52Cis 0 (zero). Accordingly, a value of the stay point associated with theidentifier a01 in the intermediate table illustrated in FIG. 53C is 5(=0+5). It should be noted that the storage unit 74 d does not store theidentifier b01 of the person detection area b1 as the comparison sourceof the currently-processed captured-image data in the intermediate tableillustrated in FIG. 53C. Instead, the storage unit 74 d keeps storingthe identifier a01 of the person detection area a1 as the comparisondestination, which is determined to be similar to the comparison source.When comparing FIG. 52A with FIG. 53A, the person detection area a1indicated the identifier a01 is determined to be the same as or similarto the person detection area b1 indicated by the identifier b01. Then,the operation returns to S321.

The processing of S321 to S329 is repeated as above, and thereby theaddition of the stay points of the identifiers 01 and a02 ends in theintermediate table as illustrated in FIG. 52C.

By contrast, when the determination unit 74 c determines that thedistance and the ratio of size are not within (outside of) thepredetermined values (S326: NO), the storage unit 74 d temporarilystores a similarity determination result as illustrated in FIG. 53B. Forexample, when no person detection area as the comparison destination isnot the same as or similar to the person detection area b3 (see FIG.53A) as the comparison source indicated by the identifier b03, thestorage unit 74 d stores “NO” in the similarity determination result forthe identifier b03. It should be noted that any information other than“NO” may be stored, provided that the information indicates that noperson detection area that is the same or similar is present. With theoperation described heretofore, in FIGS. 52A and 53A, the persondetection area b1 is determined to be the same as or similar to theperson detection area a1, and, the person detection area b2 is the sameor similar to the person detection area a2. By contrast, the persondetection area b3 and the person detection area b4 are determined not tobe the same or similar to any person detection area. In this case, inthe intermediate table illustrated in FIG. 53C, the new identifiers b03and b04 are managed in association with zero as the stay point.

By contrast, when the determination unit 74 c determines that thecaptured-image data item of the previous capturing date and timeillustrated in FIG. 52B contains no next person detection area that isclose to the person detection area as the comparison source (S325: NO),the storage unit 74 d moves the identifier indicating the persondetection area as the comparison source from the intermediate table tothe final table as illustrated in FIG. 53B. For example, the persondetection areas a3 and a4 respectively indicated by the identifiers a03and a04 in the intermediate table illustrated in FIG. 52C are not thesame as or the similar to each of the person detection areas in thecaptured-image data item of the next capturing time, the storage unit 74d moves the identifiers a03 and a04 to the final table as illustrated inFIG. 53D. Specifically, the storage unit 74 d moves, to the final tableillustrated in FIG. 53D, the identifiers and the stay points in theintermediate table (see FIG. 52C), and information indicating theposition of the center point and the size associated with the sameidentifiers in the person specification information (see FIG. 52B).

The processing of S314 to S331 is repeated as above, and in a case inwhich four person detection areas are present in a captured-image dataitem of a third capturing time as illustrated in FIG. 54A, for example,the person specification information is as illustrated in FIG. 54B.Further, in this case, the intermediate table and the final table managedata as illustrated in FIG. 54C and FIG. 54D, respectively. In thiscase, a person detection area pc1 indicated by an identifier c01 isdetermined to be the same as or the similar to the person detection areapa1 indicated by the identifier a01 in the captured-image data item ofthe second previous capturing time. Accordingly, in the intermediatetable illustrated in FIG. 54C, the stay point of the identifier a01 is11, which is obtained by adding 6 to 5.

It should be noted that the person detection areas c1, c2, c3, and c4contain the center points pc1, pc2, pc3, and pc4, respectively. Further,the person detection areas c1, c2, c3, and c4 are identified by theidentifiers c01, c02, c03, and c04, respectively.

By contrast, when the determination unit 74 c determines that there isno captured-image data item that has not yet been read out in the memory7000 (S314: NO), the storage unit 74 d moves the identifiers indicatingall of the person detection areas as the comparison sources in theintermediate table illustrated in FIG. 54C to the final table asillustrated in FIG. 55. In this case, the storage unit 74 d moves, tothe final table illustrated in FIG. 55, the identifiers and the staypoints in the intermediate table (see FIG. 54C), and informationindicating the position of the center point and the size associated withthe same identifiers in the person specification information (see FIG.54B).

With the processing steps as described heretofore, when the calculationof the stay point is performed on the person detection areas in all ofthe captured-image data items, the rendering unit 74 h renders the heatmap image based on the final table illustrated in FIG. 55. FIG. 56 is aview illustrating the predetermined-area image to which the heat mapimage generated based on the stay points is attached. FIG. 56illustrates an example image obtained by capturing the interior of anapparel store. This captured-image data including the predetermined-areaimage is the captured-image data item that is first read out at S314.The display control unit 73 maps the heat map image on thepredetermined-area image and displays the predetermined-area image onwhich the heat map image is mapped on the display 508 of thecommunication terminal 7. In the image illustrated in FIG. 56, the heatmap is actually expressed depending on the duration of stay such thatred, orange, yellow, yellow-green, and blue spots are rendered in adescending order of the duration of stay. However, in the exampleillustrated in FIG. 58, the heat map image is expressed by the gradationof gray color, such that the longer the duration of stay, the darker thecolor. This enables the viewer Y to recognize in which sales item avisitor is interested.

In a case in which a first person is present in a captured-image dataitem of a capturing time that is precedent to a capturing time of acaptured-image data item as a current processing target, and a secondperson is present in the captured-image data item as the currentprocessing target, when the determination unit 74 c determines that thefirst person and the second person are a same object, the rendering unit74 h may render a spot in light color. By contrast, when thedetermination unit 74 c determines that the first person and the secondperson are not the same object, the rendering unit 74 h may render aspot in dark color. Further, the longer a time difference between thecapturing time of the second person and the capturing time of the firstperson, the darker the color that the rendering unit 74 h may render.

Furthermore, in FIG. 56, the display control unit 73 displays a slider sbelow the captured image on which the heat map image is superimposed.The slider s specifies a threshold of the duration of stay such that aperson who stays in a place for equal to or more than the threshold isexpressed as the heat map image. FIG. 56 illustrates an example in whicha setting is configured such that a person who stays in a place for 10seconds or more is displayed by the heat map image. In other words, aperson corresponding to the identifier of which stay point is 10 or moreis displayed by the heat map image.

Still further, in FIG. 56, the display control unit 73 displays an upperlimit value (MAX) indicating a setting of an upper limit of thethreshold (duration). FIG. 56 illustrates an example in which the upperlimit value is set to 1 minute. The setting of the upper limit valueprevents a person such as a sales clerk who often stays in a same placefor a long period in order to fix sales items, etc., from beingdisplayed, and thereby only visitors are displayed by the heat mapimage.

As described heretofore, according to the fourth embodiment, the heatmap image is rendered using a specific position in each area where aperson is detected and a size of each area for the weighting ofgradation of color. Accordingly, an analysis is performed moreaccurately on how an object such a person stays in one place.

In the fourth embodiment described heretofore, although a description ismade of the detection are of a person, an object to be detected may bean animal or a pack, provided that it is a moving object. A person, ananimal and pack are examples of an object.

A recording medium such as a CD-ROM storing the programs in the aboveembodiment and the HD 504 storing those programs may be distributeddomestically or internationally as a program product.

Further, in alternative to the laptop computer, the communicationterminal 7 may be implemented by a personal computer such as a desktopcomputer, tablet, a smartphone, a tablet, or a smartwatch.

Fifth Embodiment

Hereinafter, a description is given of a fifth embodiment of the presentinvention. An operation of generating a full spherical panoramic image,an overview of a configuration of the image communication system 10, thehardware configurations of the image capturing device 1, thecommunication terminals 3 and 7, and the image management system 5according to the fifth embodiment are the same as those of the firstembodiment. Therefore, redundant descriptions thereof are omitted below.

FIG. 57 is a view illustrating an example of an analysis informationscreen 601 displayed by the communication terminal 7 according to thefifth embodiment. The analysis information screen 601 includes a fullspherical image 602. A plurality of types of analysis information arevisually displayed on the full spherical image 602. One example of theplurality of types of analysis information is a total number of persons.The total number of persons is indicated in different colors dependingon its number. The total number of persons indicates a number of personsdetected in a plurality of the full spherical images captured atdifferent times. A portion where the total number of persons isindicated in different colors is referred to as a “color-codedindication 603”.

The other example of the plurality of analysis information is a numberof individual persons. The number of individual persons indicates anumber obtained by counting a same person as one person even when thesame person is detected multiple times. In the analysis informationscreen 601, the number of individual persons is indicated by a height ofa bar chart 604. The bar chart 604 is counted on an area-by-area basis.A detailed description is given later of this area. Further, the fullspherical image 602 has coordinates in a three-dimensional space. Thebar chart also has three-dimensional information. In other words, thebar chart 604 is represented in a three-dimensional space. Because thebar chart 604 is represented in a three-dimensional space, when a viewerrotates the full spherical image 602 as described later, the viewer canview the bar chart 604 from different angles. Accordingly, the viewercan recognize in which area the bar chart 604 is present as well as therelation between the bar chart 604 and the color-coded indication 603.

Thus, the communication terminal 7 according to the fifth embodiment isable to not only display the analysis information in a color-codedindication but also display the analysis information in a stereoscopicindication as the full spherical image 602 is displayed in athree-dimensional space. Accordingly, the viewer can easily recognizethe relation between the two types of analysis information. Further,because the color-coded indication and the stereoscopic indication aredistinguishable against each other, both can be displayed together.Accordingly, it is not necessary to display the analysis information inthe color-coded indication and the analysis information in thestereoscopic indication separately with each other.

FIG. 58 is a block diagram illustrating functional configurations of theimage capturing device 1, the communication terminal 3, the imagemanagement system 5, and the communication terminal 7, which constitutea part of the image communication system 10 according the fifthembodiment.

The functional configuration of the image capturing device 1 is the sameor substantially the same as that of the first embodiment described withreference to FIG. 13. Therefore, a redundant description thereof isomitted below.

The communication terminal 3 according to the fifth embodiment includesthe data exchange unit 31, an acceptance unit 32, the connection unit38, and the data storage/read unit 39. These functional blocks 31, 32,38 and 39 are implemented by one or more hardware components illustratedin FIG. 10, when operating in accordance with instructions from the CPU301 executing according to the programs for the communication terminal3, loaded to the RAM 303 from the EEPROM 304.

The communication terminal 3 further includes the memory 3000, which isimplemented by the ROM 302 and the RAM 303 illustrated in FIG. 10. Thememory 3000 stores the programs for the communication terminal 3.

It should be noted that functions implemented by the data exchange unit31, the connection unit 38, and the data storage/read unit 39 are thesame or substantially the same as those of the first embodiment.Therefore, redundant descriptions thereof are omitted below.

The acceptance unit 32 of the communication terminal 3 is implemented bythe CPU 301 illustrated in FIG. 10, when executing according to theprogram loaded to the RAM 303, to accept an instruction input accordingto a user (an installer X in FIG. 8) operation.

The image management system 5 according to the fifth embodiment includesthe data exchange unit 51, the detection unit 52, a screen generationunit 56, a thumbnail generation unit 57, and the data storage/read unit59. These units are functions that are implemented by or that are causedto function by operating any of the hardware elements illustrated inFIG. 11 in cooperation with the instructions of the CPU 501 according tothe program for the image management system 5 expanded from the HD 504to the RAM 503.

The image management system 5 further includes the memory 5000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11.Further, the memory 5000 includes a location management database (DB)5002, an image capturing management DB 5003, an image management DB5004, a thumbnail management DB 5005, an analysis information managementDB 5006, and a score table and drawing point table DB 5007. Hereinafter,a description is given of each of the databases 5002 to 5007.

TABLE 1 AREA AREA LOCATION LOCATION LOCATION ID NAME ID NAME LAYOUT MAPTERMINAL ID a001 Tokyo s001 Shibuya-store 100008ifauy.jpg t0001, t0002,t0003, t0005, t0006 a001 Tokyo s002 Yurakucho- 100009ifauy.jpg t0021,t00022, venue t0023, t0024 . . . . . . . . . . . . . . . . . .

Table 1 is an example of a location management table that indicatesinformation stored in the location management DB 5002 in the form of atable. The location management table stores an area ID, an area name, alocation ID, a location name, a location layout map, and a device ID inassociation with one another. A line in the location management tablemay be referred to as a record hereinafter. The same applies to theother tables described hereafter. The area ID is identificationinformation for identifying an area. The area ID is, for example, acombination of an alphabet and numbers that do not overlap with oneanother.

The area name indicates a certain domain such as Tokyo, Shibuya-ku, NewYork State, and New York City. The area name may be a region name. Itshould be noted that the identification information is a name, a symbol,a character string, a numerical value, or a combination of at least twoof these items for identifying a specific object from among a pluralityof objects. The same applies to the IDs or identification informationdescribed hereinafter.

The location ID is an example of identification information foridentifying a location. The different location IDs are assigned to thedifferent location names. The location ID may refer to informationspecific to a location. The location ID is, for example, a combinationof an alphabet and numbers that do not overlap with one another. Thelocation means a place where the image capturing device 1 is placed tocapture objects surrounding the image capturing device 1. Examples ofthe location include a store.

The location name is a name of the location. Examples of the locationname include a store name such as Shibuya store, or a venue name such asShibuya venue. In the location layout map, a file name of image data,for example, indicating a layout of each location and a map isregistered. A position of the image capturing device 1 or a sales itemin the location is specified by a two-dimensional coordinate using thelocation layout map.

The device ID is identification information for identifying the imagecapturing device 1. The device ID may refer to information specific tothe image capturing device 1. Examples of the device ID include a serialnumber, a production serial number, a numerical value that does notoverlap with a model number, an IP address and a MAC address. These arejust examples, and the device ID is not limited thereto. As indicated inTable 1, more than one image capturing device 1 are placed in onelocation. The position of each of the image capturing devices 1 isregistered in the location layout map.

The installer X or the viewer Y may register the location managementtable. Alternatively, a supplier of the image communication system 10may register the location management table.

TABLE 2 CAPTURING CAPTURING LOCATION CAPTURING START DATE END DATE IDTITLE AND TIME AND TIME s001 Weekend sale 2015.7.4.10:00 2015.7.4.12:00s002 Foundation 2015.7.4.18:00 2015.7.4.09:00 anniversary . . . . . . .. . . . .

Table 2 is an example of an image capturing management table thatindicates information stored in the image capturing management DB 5003in the form of a table. The image capturing management table store acapturing title, a capturing start date and time, a capturing end dateand time in association with each location ID. The capturing title is atitle of an event. The capturing title is input by the viewer Y. Inother words, the capturing title is named after a name of a certainevent that is held in a store, for which the viewer Y wants to monitorthe behavior of consumers. However, the capturing title may not be thename of the event, because the viewer Y can assign (input) a desiredtitle. For example, the capturing title may be just a character stringindicating a capturing date and time. The viewer Y refers to thecapturing title to extract a desired captured image item from among aplurality of captured image data items. It should be noted that aplurality of image data items are obtained during a single event bycontinuously (periodically) capturing images. The capturing start dateand time is input by the viewer Y. The capturing start date and timeindicates a date and time at which the image capturing device 1 is tostart (or started) image capturing. The capturing end date and time isinput by the viewer Y. The capturing end date and time indicates a dateand time at which the image capturing device 1 is to stop (or stopped)image capturing. The viewer Y may register the capturing start date andtime and the capturing end date and time in advance such that the imagecapturing device 1 starts and ends image capturing according to aschedule registered in advance. The image management system 5 mainlyregisters the image capturing management table.

TABLE 3 CAPTURING TERMINAL IMAGE FILE NAME OF DATE ID ID IMAGE DATA ANDTIME t0001 P0001 P0001.jpeg 2015.7.4.10:00 t0001 P0002 P0002.jpeg2015.7.4.10:10 . . . . . . . . . . . .

Table 3 is an example of an image management table that indicatesinformation stored in the image management DB 5004 in the form of atable. The image management table stores an image ID, a file name ofimage data, and a capturing date and time in association with eachdevice ID. The image ID is an example of identification information foridentifying image data. The image ID may refer to information specificto an image data item. The file name of image data indicates a file nameof the image data identified by the image ID. The capturing date andtime indicates a date and time at which the image data is captured bythe image capturing device 1 identified by the device ID. The image dataare also stored in the memory 5000.

For example, a user (the viewer Y) operating the communication terminal7 accesses the communication terminal 7 to select a desired locationname and a desired capturing title from the image capturing managementtable of Table 2. The image management system 5 reads out the device IDthat is associated with the selected location ID from the locationmanagement table of Table 1. The image management system 5 specifiesimage data items whose capturing date and time are within the periodfrom the capturing start date and time to the capturing end date andtime from among image data items associated with the device ID in theimage management table.

The viewer Y may directly select a desired device ID or a desiredlocation ID. In the fifth embodiment, a description is given of a casein which the viewer Y selects a desired device ID and views, in order tosimplify the description. It should be note that the image managementsystem 5 mainly registers the image management table.

TABLE 4 FILE NAME OF IMAGE THUMBNAIL THUMBNAIL PREDETERMINED- ID IDIMAGE DATA AREA INFORMATION P0001 th123456 P0001.jpeg rH234, rV450,angle 32 P0002 th123457 P0002.jpeg rH134, rV350, angle 35 . . . . . . .. . . . .

Table 4 is an example of a thumbnail management table that indicatesinformation stored in the thumbnail management DB 5005 in the form of atable. Thumbnails owe their name to a human thumb, and a thumbnail imagerefers to image data that is reduced in size or number or pixels orimage data of multiple images to be viewed on a screen simultaneously.

The thumbnail management table stores a thumbnail ID, a file name ofthumbnail image, and the predetermined-area information in associationwith each device ID. The thumbnail ID is an example of identificationinformation for identifying a thumbnail image generated based on theimage data identified by the image ID. The thumbnail ID may refer toinformation specific to a thumbnail image. The file name of thumbnailimage is a file name of the thumbnail image identified by the thumbnailID. The image management system 5 assigns the file name of thumbnailimage. The predetermined-area information indicates a predetermined areafor which the thumbnail image is generated in the image data identifiedby the image ID. The image management system 5 mainly registers thethumbnail management table.

TABLE 5 IMAGE ID AREA ID AREA RANGE P0001 area1 x1, y1, width1, height1area2 x2, y2, width2, height2 . . . . . . P0002 area1 x1, y1, width1,height1 . . . . . . . . . . . . . . . P000n area1 x1, y1, width1,height1

Table 5 is an example of an analysis information table that indicatesinformation stored in the analysis information management DB 5006 in theform of a table. The analysis information table stores an area ID and anarea range in association with each image ID. The image ID is the sameor substantially the same as that of Table 3, and therefore a redundantdescription thereof is omitted below. The area range is information forspecifying a person detection area in image data. The person detectionarea is a rectangle that passes through some points of the contour of avisitor (person) that is detected by analyzing the image data. The areaID is an example of identification information for identifying theperson detection area. The area ID may refer to an area number orinformation specific to an area. For example, the area ID is indicatedby “area N”, where N is serial numbers assigned to each image.

The person detection area is specified based on the area range. Assumingthat the person detection area where the visitor is detected has a shapeof rectangle, for example, the area range is indicated by a coordinate(x, y) of an upper-left corner, a width and a height of the rectangle.The person detection area may be specified based on the coordinates oftwo points of diagonal corners. It should be noted that the area rangeis determined based on coordinates in the image data in a state of planebefore the image is attached to a full sphere. In other words, the imagecapturing device 1 outputs a planer image at first. The planer image isattached to the full sphere for viewing, and thereby the full sphericalimage is generated.

The score table and drawing point table DB 5007 stores the score tableand the rendering point table. The score table and the drawing pointtable according to the fifth embodiment are the same or substantiallythe same as those of the first embodiment, described above withreference to FIGS. 16A and 16B. Accordingly, the redundant descriptionsthereof are omitted below.

The data exchange unit 51 of the image management system 5 isimplemented by the network I/F 509 illustrated in FIG. 11, whenoperating under control of the CPU 501. The data exchange unit 51exchanges data with the communication terminal 3 or the communicationterminal 7 via the communication network 9. Hereinafter, a phrase “viathe data exchange unit 51” may be omitted even when the image managementsystem 5 communicates data with the communication terminal 7.

The detection unit 52 is implement by the CPU 501 illustrated in FIG.11, when executing according to the program loaded to the RAM 503. Thedetection unit 52 detects a person from the image data. This detectionof person is performed based on a suitable setting of a feature amountand a setting of a learning identification machine. Examples of thefeature amount include Haar-like features, local binary patterns (LBP)features, and histogram of oriented gradients (HOG) features. Examplesof the learning identification machine include deep learning, SVM, and acascade classifier using adaptive boosting (AdaBoost). However, anyother suitable feature amounts or learning identification machine may beused, provided that they can detect a person.

The thumbnail generation unit 57 is implement by the CPU 501 illustratedin FIG. 11, when executing according to the program loaded to the RAM503. The thumbnail generation unit 57 generates a thumbnail image of apredetermined-area image.

The screen generation unit 56 generates screen information based onwhich the communication terminal 7 displays image data when the imagedata is transmitted to the communication terminal 7. The screeninformation is described in HyperText Markup Language data, JavaScript(registered trademark), or cascading style sheets (CSS), for example.Any other suitable format may be used for generating the screeninformation, provided that the communication terminal 7 can interpretinformation.

The data storage/read unit 59 is implement by the HDD 505, whenoperating under control of the CPU 501, to store data in the memory 5000and read out data from the memory 5000. Hereinafter, a phrase “via thedata storage/read unit 59” may be omitted even when the image managementsystem 5 performs reading or writing of data from or to the memory 5000.

The communication terminal 7 includes the data exchange unit 71, thereception unit 72, the display control unit 73, the heat map generationunit 74, a bar chart generation unit 75, and the data storage/read unit79. These functional blocks 71 to 75, and 79 are implemented by one ormore hardware components illustrated in FIG. 11, when operating inaccordance with instructions from the CPU 501 executing according to theprograms for the communication terminal 7, loaded onto the RAM 503 fromthe HD 504.

The communication terminal 7 further includes the memory 7000, which isimplemented by the RAM 503 and the HD 504 illustrated in FIG. 11. Thememory 7000 stores the programs for the communication terminal 7.Examples of the programs for the communication terminal 7 includebrowser software and application software having communicationcapability such as browser software. Further, information described inHTML or a script language that is sent from the image management system5 to the communication terminal 7 is also the program for thecommunication terminal 7.

It should be noted that functions implemented by the data exchange unit71, the reception unit 72, the heat map generation unit 74, and the datastorage/read unit 79 are the same or substantially the same as those ofthe first embodiment. Therefore, redundant descriptions thereof areomitted below.

The display control unit 73 is implemented by the CPU 501 illustrated inFIG. 11, when executing according to the program loaded onto the RAM503. The display control unit 73 interprets the screen informationtransmitted from the image management system 5 to display variousscreens on the display 508 of the communication terminal 7. The displaycontrol unit 73 further includes an image display controller 81, animage superimposition unit 82, and a bar chart display controller 83.The image display controller 81 attaches the planer image data to thefull sphere to place the image data a three-dimensional space. The imagesuperimposition unit 82 superimposes the heat map image generated by theheat map generation unit 74 on the planer image data. The bar chartdisplay controller 83 places the bar chart 604 outside of the fullsphere to place the bar chart 604 in a three-dimensional space.

The bar chart generation unit 75 is implemented by the CPU 501, whenexecuting according to the program loaded to the RAM 503, to generatethe bar chart 604 as the analysis information. The bar chart generationunit 75 determines whether a first person who is detected in a certainarea in a first spherical image item and a second person who is detectedin the same certain area in a second full spherical image item that iscaptured at a different time from the first spherical image is the sameperson. The bar chart generation unit 75 determines whether the firstperson and the second person is the same person based on a combinationof at least two of a face, a dress and a size of the person. When thedetermination is performed based on a person's face, the bar chartgeneration unit 75 performs face recognition to specify the position ofa face an eyebrow, an eye, a nostril, a lip, etc. The bar chartgeneration unit 75 determines whether the two persons are the sameperson based on a distance between any two of these parts or an angle ofa straight line connecting any two of these parts. When thedetermination is performed based on a dress that the person is wearing,the bar chart generation unit 75 generates a histogram of pixel value.The bar chart generation unit 75 uses the histogram to determine whetherthe two persons are the same person based on a ratio of the number ofpixels of each pixel value. When the determination is performed based ona person's size, the bar chart generation unit 75 determines whether aheight of a rectangle that passes through some points of the contour ofthe first person is the same as that of the second person to determinewhether the two persons are the same person.

The bar chart generation unit 75 counts the number of individual personson an area-by-area basis. In other words, the bar chart generation unit75 distinguishes one person from another person, and counts the numberfor each of the same persons. When a different person is detected, thebar chart generation unit 75 counts the number of persons for eachperson. Accordingly, the number of individual persons is counted on anarea-by-area basis for each of the individual persons. Even when thenumber of the individual persons counted for the same person is two ormore, it is regarded as one person. Further, the bar chart generationunit 75 calculates a duration of stay in an area for the individualperson. For example, in a case in which the same person is detected inthree images that are captured continuously, the duration of stay is 3.The duration of stay may be converted to a time period based on aninterval of image capturing or a capturing time as metadata of the fullspherical image.

An operation of generating the heat map image according to the fifthembodiment is the same or substantially the same as that of the firstembodiment described above with reference to FIGS. 17 to FIGS. 27A and27B. Therefore, a redundant description thereof is omitted below.

The heat map image as illustrated in FIG. 28 indicates a place where aperson is likely to be detected in an intense color such as red.Accordingly, the viewer Y can recognize a place where a person is likelyto be detected at a glance. However, it is difficult for the viewer Yviewing the heat map image to tell whether one person is detectedmultiple times at a same place or each of plural persons is detectedonce.

To address this issue, in the fifth embodiment, the communicationterminal 7 determines a person detected in a certain image data itemcaptured at a certain time and a person detected in a different imagedata item captured at a different time are the same person to count thenumber of individual persons. Further, the communication terminal 7indicates the number of individual persons by the bar chart 604.

FIG. 59 is an illustration for explaining drawing of the bar chart 604.In order to draw the bar chart 604, the bar chart generation unit 75divides the planer image data into a plurality of areas 610. The area610 has a square or a rectangular shape. The size of the area 610 is notparticularly limited, and the viewer Y may set the size as desired, forexample. In a case in which the viewer Y does not set the size, the barchart generation unit 75 divides the image into the areas 610 b at adefault size.

The bar chart generation unit 75 calculates how many different personsare detected for each of the areas 610 in the full spherical images thatare captured within a period of time that is specified by the viewer Y,for example. For example, it is assumed that a person A is detected forthree times and a person B is detected once in a certain area 610 (it isnot necessary to perform personal identification, because it issufficient to tell whether the two person are the same person). In thiscase, the number of detections is 3 for the person A, and the number ofdetections is 1 for the person B. However, the number of individualpersons is 1 for the person A, and the number of individual persons is 1for the person B. By contrast, in the heat map image, the color-codedindication 603 indicates that the four persons are detected.Accordingly, in a case in which a same person is detected multiple, aviewer Y viewing the bar chart 604 in comparison with the heat map imagecan recognize that the color is red because the same person is detectedmultiple times. By contrast, in a case in which the bar chart 604 islong, the viewer Y recognizes that the color of the heat map image isred because many different persons are detected.

FIG. 60 is a flowchart illustrating an operation of generating the barchart 604 by the bar chart generation unit 75. The operation of FIG. 60starts in response to an instruction for displaying the bar chart 604 bythe viewer Y.

The bar chart generation unit 75 acquires a resolution of the planerimage data and the analysis information stored in the analysisinformation management DB 5006 (S401).

Next, the bar chart generation unit 75 acquires a group of images (aseries of image data) (S402). The series of images data are a pluralityof image data items that are captured by one image capturing device 1during a period of time that is designated by the viewer Y as describedlater with reference to FIG. 62. Alternatively, the series of image dataare all of image data items captured by a certain image capturing device1.

The bar chart generation unit 75 refers to the image data items one byone (S403).

The bar chart generation unit 75 determines whether a next persondetection area is present based on the analysis information stored inthe analysis information management DB 5006 (S404).

When the bar chart generation unit 75 determines that the next persondetection area is not present (S404: NO), the operation proceeds toS408. By contrast, when the bar chart generation unit 75 determines thatthe next person detection area is present (S404: YES), the bar chartgeneration unit 75 determines whether a person in this person detectionarea was previously detected in a same area 610 (S405). Specifically,the bar chart generation unit 75 first determines the area 610containing a barycenter (or a center) of the person detection area.Next, the bar chart generation unit 75 determines whether there is aperson detected in the same area 610 in an image data item captured at aprevious capturing time. When the bar chart generation unit 75determines that there is a person detected in the same area 610, the barchart generation unit 75 determines whether a person in the area rangein the currently-processed image data item and a person in the arearange in the captured image item that is captured at the previouscapturing time (two persons in the person detection areas whosebarycenter or center are contained in the same area 610) are the sameperson. The determination at S405 is performed as above. When the twoperson is the same person, the determination result at S405 is YES.

When the determination result of S405 is YES, the bar chart generationunit 75 increments the number of detections of the individual person forthe area 610 containing the barycenter or center of the person detectionarea (S406). For example, the person A is detected in the same area 610,the number of detections in this area 610 for the person A isincremented by one. The number of detections of the person A isincremented as above in order to measure a duration of stay of theperson A. Accordingly, it is not necessary to increment the number ofdetections when just measuring the number of the individual persons forthe person A.

When the determination result at S405 is NO, the bar chart generationunit 75 records information indicating that a person is detected in thearea A containing the barycenter or center of the person detection area(S407). For example, when the person B is first detected in the area 610containing the barycenter or center of the person detection area, thebar chart generation unit 75 records information indicating that aperson is detected in this area 610. The information may not includeinformation for identifying the person B.

Next, the bar chart generation unit 75 determines whether a next imagedata item is present (S408). When there is a next image data item (S408:YES), the operation returns to S403 to repeat S403 to S407.

When there is no next image data item (S408: NO), the bar chartgeneration unit 75 calculates a sum of the numbers of individual personsfor on an area-by-area basis (S409). In other words, even when theperson A is detected multiple times in the same area 610, the number ofindividual persons is counted as 1. Further, when the person B isdetected even once in the area 610, the number of individual persons iscounted as 1. Accordingly, the number of individual persons for thisarea 610 is 2 in total. It should be noted that the total number may notbe necessarily calculated. In this case, the number of individualpersons is stored on a person-by-person basis.

Next, the bar chart generation unit 75 calculates a duration of stay foreach of the detected persons (S410). In other words, the bar chartgeneration unit 75 calculates how long a same person stayed in the area610. For example, in a case in which the person A is detected for threetimes in a same area 610, the duration of stay is 3. Further, forexample, in a case in which the person B is detected once in the area610, the duration of stay is 1. These numerical values 3 and 1 as theduration of stay may be converted to a period of time such as minutes.The duration of stay may be summed on a person-by-person basis.Alternatively, the duration of stay may be summed on an area-by-areabasis.

A description is given heretofore of a case in which the bar chartgeneration unit 75 determines whether the same person was detected inthe image data item that is captured at the previous period of time.However, the bar chart generation unit 75 may determine that the sameperson was detected in all of the image data items that are capturedbefore the capturing date and time of the image data item as a currentprocessing target. The bar chart generation unit 75 counts the number ofindividual persons for each same person. Accordingly, a person who onceleft a certain sales place and come back later to the sales place can beidentified. Further, the bar chart generation unit 75 is able to countthe number of persons who come back to the place. For example, assumingthat the person A is detected for three times in total, in a case inwhich the person A is detected once after he/she comes back to a salesplace from among the three detected times, a number of times that theperson A comes back is one. The number of times that a person comes backmay be counted for other visitors such as the person B in substantiallythe same manner.

Next, the bar chart display controller 83 renders the number ofindividual persons on an area-by-area basis (S411). A detaileddescription is given later of the rendering performed at S411 withreference to FIGS. 61A and 61B. It should be noted that in the fifthembodiment, the rendering means displaying a three-dimensional bar charton the two-dimensional display 508.

The operation of FIG. 60 may be performed by the image management system5 instead of the communication terminal 7. In this case, settings inputby the viewer Y on a setting screen illustrated in FIG. 62 istransmitted to the image management system 5. The image managementsystem 5 generates the heat map image and superimposes the heat mapimage on image data. Further, the image management system 5 counts thenumber of the individual persons to generate the bar chart 604, andarranges the bar chart 604 on the full spherical image 602. Furthermore,the image management system 5 renders this image on a two-dimensionalscreen and transmits the screen to the communication terminal 7.

Hereinafter, a description is given of the rendering of the bar chart604 with reference to FIGS. 61A and 61B. The bar chart displaycontroller 83 displays the bar chart 604 illustrated in FIG. 61Aindicating the total number of the individual persons for each area 610on the full spherical image 602 as illustrated in FIG. 61B. The barchart 604 may indicate the total duration of stay for each area 610.

To simplify the description, the bar chart display controller 83arranges the planar image data in a three-dimensional local coordinatesystem, and generates the bar chart 604 in the area 610. First, becausea center P of each area 610 is known, a position P′ (x₀, y₀, z₀) in FIG.61B is calculated in substantially the same manner as the conversionfrom the planar image data to the full spherical image.

An equation of a tangent plane 620 that contact the full sphere at theposition P′ is given as below. a, b, c, are coordinates of the center ofthe full spherical image 602. The coefficients of x, y, and z are anormal vector n of the tangent plane 620.(x0-a)(x-a)+(y0-b)(y-b)+(z0-c)(z-c)=R²

As illustrated FIG. 61A, the bar chart display controller 83 generatesthe bar chart 604 whose height is proportional to the number of thenumber of individual persons in each area 610 in the local coordinatesystem. To simplify the description, it is assumed that the directionsof x, y, and z axes of the local coordinate system is the same as thoseof the world coordinate system. When arranging the bar chart 604 of thelocal coordinate system in the world coordinate system, the bar chartdisplay controller 83 rotates the bar chart 604 of the local coordinatesystem by an angle defined by (x₀-a) and the x-axis, an angle defined by(y₀-b) and the y-axis, and an angle defined by (z₀-c) and the z-axis.Further, the bar chart display controller 83 moves the center point P tothe position P′ by parallel displacement.

Thus, the three-dimensional bar chart 604 is attached to the outersurface of the full spherical image 602. Because the bar chart 604 isarranged in the world coordinate system, the display control unit 73displays the bar chart 604 on the display 508. It should be noted thatthe duration of stay can be also displayed by the bar chart 604 insubstantially the same manner as the number of individual persons.

As described heretofore, because the communication terminal 7 accordingto the fifth embodiment displays the number of individual personsthree-dimensionally on the full spherical image, it is able to displaythe analysis information in a color-coded indication together with theanalysis information in a stereoscopic indication. This makes it easierfor a viewer to grasp the relation among the plurality types of analysisinformation.

As described heretofore, a size of analysis information is indicated bya shape (height) of the bar chart 604. However, the height is just anexample, and the thickness of the bar chart 604 may be used to indicatethe size of analysis information. Further, any other suitablethree-dimensional object than the bar chart 604 may be used forindicating the analysis information, provided that it can indicate theanalysis information three-dimensionally. For example, a polygonalcolumn such as a cylinder and a triangular prism or a polygonal pyramidsuch as a cone and a triangular pyramid may be used for indicating theanalysis information.

A description is given heretofore of an example in which the analysisinformation include the total number of persons, the number ofindividual persons, and the duration of stay, and the total number ofpersons is indicated by the color-coded indication 603 while the numberof individual persons or the duration of stay is indicated by the barchart 604. However, these are just examples of the analysis information.Other examples of the analysis information are described below.

TABLE 6 COLOR-CODED INDICATION BAR CHART INDICATION Total number ofperson Number of individual persons Number of individual persons Numberof persons who bought a sales item in an area Number of individualpersons Sales amount in an area Number of persons who pick Number ofpersons who bought a up a sales item sales item in an area

Table 6 represents examples of a combination of the analysis informationindicated by the gradation of color (color-coded indication 603) and theanalysis information indicated by the bar chart 604. The analysisinformation indicated by the gradation of color is an example of firstanalysis information. The analysis information indicated by the barchart 604 is an example of second analysis information.

First, in a combination of the number of individual persons and thenumber of persons who bought an item in an area, the number ofindividual persons is displayed by the gradation of color. In this case,the heat map generation unit 74 displays the number of individualpersons by the gradation of color (generate the heat map image) byperforming substantially the similar operation described above withreference to FIGS. 59 to FIGS. 61A and 61B because the difference isjust the replacement of the total number of individual persons to thenumber of individual persons. In this case, the number of individualpersons is a sum of a plurality of the numbers of individual persons onan area-by-area basis. Alternatively, the areas 610 may be colored indifferent colors depending on the sum of the plurality of the numbers ofindividual persons. The number of persons who bought an item in an areais a number of persons who bought a sales item in each area 610. Forexample, the Point of Sales (POS) manages the amount of sales for eachsales item in real time, it has data indicating when and which items issold. Further, it is assumed that the bar chart generation unit 75associates a sales item ID with each area 610. The POS may have dataassociating the area 610 with the sales item ID. The bar chartgeneration unit 75 counts the number of sales items on a sales recordthat match the sales item ID associated with the area 610 during aperiod of time designated by the viewer Y. The number of sales itemssold in the area 610 may be regarded as the number of persons who boughta sales item. Alternatively, the number of persons may be countedinstead of the number of sales items sold.

In a combination of the number of individual persons and an amount ofsale in an area, the number of individual persons is displayed by thegradation of color. The heat map image indicating the number ofindividual persons may be the same or substantially the same as that ofthe combination of the number of individual persons and the number ofpersons who bought an item in an area. Further, the POS is used fordetermining the amount of sales in the area 610. The bar chartgeneration unit 75 calculates a sum of amounts of sales for the salesitem on a sales record that matches the sales item ID associated withthe area 610 during a period of time designated by the viewer Y.

In a combination of a number of persons who pick up a sales item and anumber of persons who bought an item in an area, the number of personswho pick up a sales item is displayed by the gradation of color. In thiscase, the image management system 5 stores in advance, in the analysisinformation management DB 5006, the area range of a person who picks upa sales item, which is obtained by image analysis. An action of pickingup a sales item is detected using machine learning or pattern matching.Further, the communication terminal 7 may analyze a person who picks upa sales item. Because the area range of the person is acquired for eachimage data, the heat map generation unit 74 is able to display thenumber of persons who pick up a sales item by the gradation of color insubstantially the same manner as that of the total number of persons.

As described heretofore, according to the fifth embodiment, varioustypes of analysis information can be visually displayed in a combinationof the color-coded indication 603 and the bar chart 604.

Hereinafter, a description is given of a user interface of thecommunication terminal 7 with reference to FIG. 62. FIG. 62 is a viewillustrating an example of a setting screen 701 displayed on the display508 of the communication terminal 7. The setting screen 701 includes animage display area 702, a heat map drop-down list 703, a bar chartdrop-down list 704, a setting box 706 for setting an upper limit and alower limit of duration of stay, a display radio button 705, a date andtime range setting box 707, and a camera ID drop-down list 708.

The full spherical image 602 is displayed in the image display area 702.The full spherical image 602 that is selected from a plurality of fullspherical images as analysis targets is displayed in the image displayarea 702. The heat map drop-down list 703 enables the viewer Y to selecta desired type of analysis information to be indicated by the gradationof color. The bar chart drop-down list 704 enables the viewer Y toselect a desired type of analysis information to be indicated by the barchart 604. The setting box 706 for setting an upper threshold and alower threshold of duration of stay enables the viewer Y to designate alower limit and an upper limit of the duration of stay. The displayradio button 705 enables the viewer Y to designate whether the heat mapimage and the bar chart 604 are to be displayed. The date and time rangesetting box 707 enables the viewer Y to designate a range of date andtime during which image data items as analysis targets are captured. Thecamera ID drop-down list 708 enables the viewer Y to designate a desiredimage capturing device 1 by which image data items as analysis targetsare captured

Because various conditions for display are displayed on the settingscreen 701, the viewer Y can switch the display condition interactively.For example, the communication terminal 7 does not display the number ofindividual persons of a person whose duration of stay is short dependingon the setting of the lower limit of the duration of stay. By contrast,the communication terminal 7 does not display the number of individualpersons of a person whose duration of stay is extremely long dependingon the setting of the upper limit of the duration of stay. Thus, thesetting box 706 for setting an upper threshold and a lower threshold ofduration of stay enables filtering of the number of individual persons,etc., indicated by the bar chart 604.

This filtering depending on the duration of time is implemented byperforming the process of S410 before the process of S409 to count thenumber of persons whose duration of stay is within a range between theupper limit and the lower limit.

It should be noted that the setting screen 701 may further include amenu for setting a size of the area 610 or the number of areas 610.

The setting screen 701 further includes a storing data key 709, animport key 710, and an export key 711. In response to a user selectionof the storing data key 709, the reception unit 72 accepts the userselection, and the heat map generation unit 74 storesanalysis-information display data in the memory 7000. Table 7 is anillustration for explaining the analysis-information display data.

TABLE 7 DATA IMAGE HEAT MAP DATA OF DURATION TARGET ID ID IMAGE HEIGHTOF STAY RANGE Q001 au12345 Total number Number of Lower limit: 0 2016Apr. 3 of persons individual Upper limit: 25 12:00 (heat001.jpg) persons— Area 1: 2016 Apr. 3 N one 16:00 person Area 1: N two persons

In the analysis-information display data of Table 7, a data ID, theimage ID, the heat map image, data of height, the duration of stay, anda target range are stored in association with one another. The data IDis an example of identification information for identifying theanalysis-information display data to be stored. The image ID is the sameor substantially the same as that described above. More specifically,the image ID in Table 7 is an image ID of an image item that is beingdisplayed on the display 508 from among the plurality of image dataitems used for the analysis. The type of analysis information indicatedby the gradation of color and data of the heat map image are stored asthe heat map image. The type of analysis information indicated by thebar chart 604 and the number of persons (e.g., the number of individualpersons in Table 7) in each area are stored as the data of height. Theupper limit and the lower limit, each being designated in the settingbox 706 are stored as the duration of time. The value that is designatedin the date and time range setting box 707 is stored as the targetrange.

Because the analysis-information display data obtained by aggregatingthe data set via the setting screen 701 is stored in the memory asdescribed above, it takes less time to display the image than it takesto aggregate the data when the viewer Y later instruct the display. Inaddition, it is not necessary to download the image data again, the costrequired for data communication and calculation is reduced.

In response to a user selection of the export key 711, the receptionunit 72 accepts the user selection, and the data storage/read unit 79stores the analysis-information display data in a predetermined fileformat in the memory 7000 or a storage medium. Examples of thepredetermined file format include an Excel format. In Excel format, theimage data, the heat map image, and the data of height are stored on asheet, and other information contained in the analysis-informationdisplay data are stored on a sheet. Accordingly, the viewer Y is able tomanipulate the analysis-information display data edited in thepredetermined file format, or edit the data of height.

In response to a user selection of the import key 710, the receptionunit 72 accepts the user selection, and the data storage/read unit 79reads out the analysis-information display data in the predeterminedfile format from the memory 7000 or a storage medium. In addition, thedata storage/read unit 79 converts the predetermined file format to adata format that the communication terminal 7 can process. For example,in a case in which the analysis-information display data is stored inExcel format, the data storage/read unit 79 has information specifying asheet and a cell in which the analysis-information display data isstored. This enables the communication terminal 7 to convert theexported analysis-information display data in the predetermined fileformat to the file format that the communication terminal 7 can process.Accordingly, in response to an instruction given by the viewer Y, thecommunication terminal 7 is able to display the data of height edited inthe predetermined file format.

The image data is attached to the inner surface of the full sphere asdescribed above. However, the image data may be attached to the outersurface of the full sphere. The communication terminal 7 switches theside to which the image data is attached based on the switchingcondition as indicated in the following Table 8, for example.

TABLE 8 POSITION OF DRAWING POSITION ANALYSIS VIRTUAL CAMERA OF IMAGEINFORMATION Full sphere is not within Draw an image on an inner Heat mapimage an angle of view surface of the sphere Full sphere is within anDraw an image on an outer Heat map image angle of view surface of thesphere Bar chart

According to the switching condition of Table 8, when the virtual camerais placed such that the entire full spherical image 602 is not includedwithin a range defined by an angle of view, the image data is attachedto the inner surface of the full sphere, and the heat map image isdisplayed as the analysis information. By contrast, when the virtualcamera is placed such that the entire full spherical image 602 isincluded within a range defined by an angle of view, the image data isattached to the outer surface of the full sphere, and the heat map imageand bar chart 604 is displayed as the analysis information.

Hereinafter, a detailed description is given of the switching of theside to which the image data is attached with reference to FIGS. 63A and63B. FIGS. 63A and 63B are example illustrations for explaining therelation between the full sphere and the position and angle of view ofthe virtual camera IC. In FIG. 63A, the entire full spherical image 602is not included in the range defined by the angle of view α of thevirtual camera IC. Accordingly, the image data is attached to the innersurface of the full sphere, and only the heat map image (color-codedindication 603) is displayed. By contrast, in FIG. 63B, the entire fullspherical image 602 is included in the range defined by the angle ofview α of the virtual camera IC. Accordingly, the image data is attachedto the outer surface of the full sphere, and both the heat map image(the color-coded indication 603) and the bar chart 604 are displayed.

This enables the viewer Y to switch the display as desired. For example,when the viewer Y wants to view a plurality types of analysisinformation together with the image, the full spherical image 602 isviewed from the outside of the sphere. By contrast, when the viewer Ywants to view the image itself enlarged, the full spherical image 602 isviewed from the inside of the sphere.

Hereinafter, a description is given of how the switching is determinedwith reference to FIG. 64A. In FIG. 64A, 2R denotes a diameter of thefull sphere, and a denotes the angle of view of the virtual camera IC.When the virtual camera IC is placed at a position away from the centerof the full sphere by a distance L₀, the entire sphere is just includedin the range defined by the angle of view α of the virtual camera IC.Distance L₀=R/(tan(α/2) The distance L between the coordinate of thevirtual camera IC in the world coordinate system and the center of thefull sphere is periodically calculated. The image superimposition unit82 and the bar chart display controller 83 determines whether DistanceL>Distance L₀ to determine the drawing position (inside or outside) ofthe image and the presence of display of the bar chart 604.

Hereinafter, a description is given of drawing the image in the insideof the full sphere and drawing the image in the outside of the fullsphere with reference to FIGS. 64B to 64E. The image superimpositionunit 82 draws the image that is drawn inside the full sphere asillustrated in FIG. 64B to the outside of the full sphere as illustratedin FIG. 64C. Alternatively, the image superimposition unit 82 performsthe drawing the other way around. Accordingly, the image is moved fromthe inside to the outside or from the outside to the inside by paralleldisplacement.

FIG. 64D is a view illustrating the full sphere viewed from top. FIG.64E is a view illustrating the full sphere viewed from side. First, itis assumed that a range from the longitude θ1 to θ2 is displayed on thedisplay 508. The longitude θ1 to θ2 can be obtained by an inverseconversion of the conversion performed when a three-dimensional objectis projected to a two-dimensional screen. In this case, a degree of thedisplayed longitude range is A degree. Although the A degree variesdepending on a latitude, the A degree is calculated based on acoordinate on the display 508. Alternatively, because the distance L₀ isconstant, the A degree with respect to the distance L₀ is obtained inadvance for each latitude.

When the inner image is moved to the outside by parallel displacement,θ1 moves to θ1′ and θ2 moves to θ2′. Accordingly, θ1′ and θ2′ areobtained by the following equations.θ1′=θ1−(180−A)θ2′=Θ2+(180−A)

Next, it is assumed that a range from the latitude φ1 to φ2 is displayedon the display 508. The latitude φ1 to φ2 can be obtained by an inverseconversion of the conversion performed when a three-dimensional objectis projected to a two-dimensional screen. In this case, a degree of thedisplayed latitude range is B degree. Although the B degree variesdepending on a longitude, the B degree is calculated based on acoordinate on the display 508. Alternatively, because the distance L₀ isconstant, the B degree with respect to the distance L₀ is obtained inadvance for each longitude.

When the inner image is moved to the outside by parallel displacement,φ1 moves to φ1′ and φ2 moves to φ2′. However, in parallel displacement,the latitude may be constant. Accordingly, the relation expressed by thefollowing equations is satisfied.φ1′=φ1φ2′=φ2

As described above, an image of a rectangular area defined by θ1, φ1, θ2and φ2 is moved to a rectangular area defined by θ1′, φ1′, θ2′ and φ2′.Because the latitude is invariant, only the longitude is changed. Animage of a range from θ1′ to θ1 and an image of a range from θ2′ to θ2also can be drawn on the outer surface of the sphere in consideration ofthe direction.

It should be noted that the bar chart 604 is displayed in substantiallythe same manner as described above by specifying the center (barycenter)of the area 610 and obtaining the normal vector. The center (barycenter)of the area 610 can be obtained by changing the longitude of the center(barycenter) of the area in substantially the same manner.

The bar charts 604 of the areas 610 may be smoothed based on the heightof each bar chart 604. FIG. 65A is a view illustrating the fullspherical image 602 including a smoothed graph 720 that is obtained bysmoothing the height of the bar charts 604. FIG. 65B is a viewillustrating just the smoothed graph 720. Even when there is an area 610in which the bar chart 604 is not present, the smoothed graph 720 givesa height to a space where the bar chart 604 is not present.

FIG. 66 is an illustration for explaining a smoothing processing. As thesimplest way, the bar chart display controller 83 processes the barcharts 604 in the image data arranged in the local coordinate system toconnect the centers of top surfaces of the bar charts 604 that areadjacent to one another by a predetermined distance or below by astraight line. One plane is formed by connecting the center points oftop surfaces of three or four bar charts 604. A polygon constituted bythis plane is arranged in a three-dimensional space to smooth the areas610. Thus, the smoothed graph 720 is displayed on the full sphericalimage 602.

Further, a smoothing filter may be used. For example, in a case in whicha filter is applied to each of 3×3 areas 610, a 3×3 square operator(kernel) is prepared in which each element is 1/9. Using the operator, aheight of an area of interest is obtained by multiplying the height ofthe area of interest by the height of each of the neighboring areas.This processing is performed on each of the areas 610 to smooth theentire height. The smoothed height is obtained in each of the areas 610,a polygon that is obtained by connecting the smoothed heights may beused for indication. Alternatively, the bar chart 604 having thesmoothed height may be used for indication. It should be noted that thesmoothing processing may be performed using a Gaussian filter in whichthe operator is determined that the closer to the area of the element,the larger the element.

The present invention is not limited to the details of the exampleembodiments described above, and various modifications and improvementsare possible.

For example, although in the fifth embodiment, one type of the analysisinformation is indicated by the color-coded indication 603 and anothertype of the analysis information is indicted by the bar chart 604, aplurality of types of the analysis information may be indicated togetherby the color-coded indication 603, or a plurality of types of theanalysis information may be indicated together by the bar chart 604.

Further, although in the fifth embodiment, the bar chart 604 is arrangedon the outer surface of the full sphere, the bar chart 604 may bearranged inside the full sphere.

Furthermore, an information processing apparatus including thecommunication terminal 7 and the image management system 5 may performthe processing described above in the fifth embodiment. Still further,the communication terminal 7 may implement at least one of the functionsimplemented by the image management system 5. The communication terminal7 may implement at least one of the functions implemented by the imagemanagement system 5. In other words, in the fifth embodiment, the imagemanagement system 5 may perform the operation that is performed by thecommunication terminal 7 in the above description, and vice versa.

The functional configurations of the image capturing device 1, thecommunication terminal 3, the image management system 5, and thecommunication terminal 7 are divided into the functional blocks asillustrated in FIG. 58, for example, based on main functions to make iteasy to understand the processing. However, the present invention shouldnot be limited by these divided processing units or the names. Theprocessing implemented by the image capturing device 1, thecommunication terminal 3, the image management system 5, and thecommunication terminal 7 may be divided to a plurality of processingunits depending on the content of processing. Further, one processingunit may include further processing units. Furthermore, the imagecommunication system 10 may include two or more image management systems5.

Still further, although a description has been given of an example inwhich the memory 5000 of the image management system 5 stores thedatabases 5002 to 5007, the databases 5002 to 5007 may be stored on thecommunication network 9 such that the image management system 5 readsout or write data from or to the databases.

It should be noted that the image superimposition unit 82 is an exampleof an information associating processor (or circuitry), the imagedisplay controller 81 is an example of a placing processor (orcircuitry), the bar chart display controller 83 is an example of ananalysis information placing processor (or circuitry), the displaycontrol unit 73 is an example of a display processor (or circuitry), thedisplay 58 is an example of a display device, and the bar chart displaycontroller 83 is an example of a generating processor (or circuitry).The reception unit 72 is an example of an acceptance processor (orcircuitry), and the memory 7000 is an example of a storage device.

According to an aspect of the present invention, an image managementsystem for managing image data to be transmitted to a communicationterminal includes: a detection unit to detect a head including a face ofa person in the image data; a calculation unit to, in a case in whichthe detection unit detects the face of person, calculate a position of afoot of the person from the position of the head in the image data basedon a size of the head including the detected face and an averageperson's body size; and a transmission unit to transmit foot positioninformation indicating the calculated position of foot.

The calculation unit may calculate the size of the head based on a topof the head and a bottom of the head.

The calculation unit may calculate a distance between a center of thehead and the foot based on the calculated size of the head and theaverage person's body size, and calculate a position that is away from aposition of the head by the distance between the center of the head andthe foot as the position of the foot in the image data.

The image management system may further include a person imagemanagement unit to manage the position information of the footcalculated by the calculation unit with image identification informationfor identifying image data containing an image of the foot, wherein thetransmission unit transmits, to the communication terminal, the imagedata, and the position information of the foot associated with imageidentification information of the image data and the imageidentification information.

The image data may be full-spherical panoramic image data.

According to another aspect of the invention, an image communicationsystem includes the image management system according to the aboveaspect and the communication terminal, wherein the transmission unit ofthe image management system transmits, to the communication terminal,respective foot position information indicating a position of a foot ofa person in a plurality of image data items that are obtained bycontinuous image capturing, wherein the communication terminal includes:a heat map generation unit to generate a heat map image indicating atrack of the person based on the respective foot position information;and a display control unit to map the generated heat map image onspecific image data from among respective image data indicated byrespective image identification information associated with therespective foot position information and control a display unit todisplay the specific image data on which the heat map is mapped.

The communication terminal may be one of a personal computer, asmartphone, a tablet terminal, and a smartwatch.

According to still another aspect of the present invention, an imagetransmission method performed by an image management system that managesimage data to be transmitted to a communication terminal, includes: adetection step of detecting a head including a face of a person from theimage data; a calculation step of, in a case in which the face of theperson is detected at the detection step, calculating a position of afoot of the person from the position of the head in the image data basedon a size of the head including the detected face and an averageperson's body size; and a transmission step of transmitting footposition information indicating the calculated position of foot.

According to still another aspect of the present invention, an imagedisplaying method performed by an image communication system thatmanages image data is provided, includes: a detection step of detectinga head including a face of a person from the image data; a calculationstep of, in a case in which the face of the person is detected at thedetection step, calculating a position of a foot of the person from theposition of the head in the image data based on a size of the headincluding the detected face and an average person's body size; and aheat map generation step of generating a heat map based on the positionof the foot calculated at the calculation step; and a display controlstep of mapping the generated heat map image on the image data andcontrolling a display means to display the image data on which the heatmap image is mapped.

According to still another aspect of the present invention, a program isprovided, that causes a computer to perform the above image transmissionmethod.

According to still another aspect of the present invention, acommunication terminal that maps a heat map image on a captured imagerepresented by captured-image data obtained by image capturing anddisplays the captured image on which the heat map image is mapped,includes a calculation unit to calculate a specific position in adetection area of each of objects in a plurality of captured-image dataitems that are obtained by image capturing with a lapse of time and asize of the detection area; a rendering unit to render a heat map imageby using the specific position and the size calculated by thecalculation unit for weighting of gradation of color; and a displaycontrol unit to map the heat map image on the captured image and displaythe captured image on which the heat map is mapped.

The communication terminal may further include a determination unit todetermine whether a first position in a detection area of a first objectin first captured-image data from among the plurality of captured imagedata items and a first size of the detection area is within apredetermined range when compared with a second position of a detectionarea of a second object in second captured-image data that is capturedafter capturing of the first captured-image data to determine the firstobject and the second object are a same object, wherein the renderingunit changes the weighting of gradation of color depending on whetherthe determination unit determines that the first object and the secondobject is the same object or the determination unit determines that thefirst object and the second object are not the same object.

In a case in which the determination unit determines that the firstobject and the second object are the same object, the rendering unit mayrender in a light color. In a case in which the determination unitdetermines that the first object and the second object are not the sameobject, the rendering unit may render in a dark color.

The longer a time difference between a capturing time of the secondobject and a capturing time of the first object, the darker color therendering unit may render.

The calculation unit may define the detection area by a position of anarbitrary corner of a rectangle area containing the object and a rangeindicating a width and a height of the rectangle area with respect tothe position of the arbitrary corner.

The communication terminal may be one of a personal computer, asmartphone, a tablet terminal, and a smartwatch.

According to still another aspect of the present invention, an imagecommunication system includes the communication terminal according tothe above aspect and an image management system that manages theplurality of captured-image data items, wherein the communicationterminal includes a reception unit to receive the plurality ofcaptured-image data items from the image management system.

According to still another aspect of the present invention, a displaymethod is provided, performed by a communication terminal that maps aheat map image on a captured image represented by captured-image dataobtained by image capturing and displays the captured image on which theheat map image is mapped. The method includes a calculation step ofcalculating a specific position in a detection area of each of objectsin a plurality of captured-image data items that are obtained by imagecapturing with a lapse of time and a size of the detection area; arendering step of rendering a heat map image by using the specificposition and the size calculated by the calculation unit for weightingof gradation of color; and a display controlling step of to mapping theheat map image on the captured image and displaying the captured imageon which the heat map is mapped.

According to still another aspect of the present invention, a program isprovided that causes a computer to perform the display method accordingto the above aspect.

According to still another aspect of the present invention, an imageprocessing system is provided, that displays image data whose angle iswider than a predetermined angle and analysis information of the imagedata. The image processing system includes an information associatingunit to associate first analysis information obtained by analyzing theimage data with the image data; a placing unit to place, in athree-dimensional space, the image data with which the first analysisinformation is associated; an analysis information placing unit to placesecond analysis information obtained by analyzing the image data as anobject in a three-dimensional space in association with the image datain the three-dimensional space; and a display unit to display the firstanalysis information, the second analysis information and the image dataon a display.

The information associating unit may indicate the first analysisinformation with gradation of color. The analysis information placingunit may indicate a size of the second analysis information with a shapeof the object.

The image processing system may further include a generation unit toanalyze the image data for each of a plurality of areas that areobtained by dividing the image data to generate the second analysisinformation for each of the plurality of areas, wherein the analysisinformation placing unit may place the object having the shaperepresenting the size of the second analysis information in the area ofthe image data in the three-dimensional space.

The placing unit may attach the image data to a sphere to place theimage data in the three-dimensional space. The analysis informationplacing unit may place the object on a surface of the sphere.

The analysis information placing unit may smooth the height of theobject in each area based on the height of objects in the area near theeach area.

The placing unit may attach the image data to the sphere to place theimage data in the three-dimensional space. The display unit may projectan image of at least a part of the sphere that is within an angle ofview of a viewpoint having a predetermined angle of view on the display.In a case in which an entirety of the sphere is within the angle ofview, the placing unit may attach the image data on an outer surface ofthe sphere. In a case in which an entirety of the sphere is not withinthe angle of view, the placing unit attaches the image data on an innersurface of the sphere.

In a case in which an entirety of the sphere is within the angle ofview, the analysis information placing unit may place the object on thesphere. In a case in which an entirety of the sphere is not within theangle of view, the analysis information placing unit may not place theobject on the sphere.

The display unit may display a setting screen that accepts an input forsettings of display of the first analysis information and the secondanalysis information on the display. The image processing system mayfurther include a reception unit for accepting a selection of the imagedata on which the first analysis information and the second analysisinformation.

The reception unit may accept a designation of a range of the secondanalysis information. The analysis information placing unit may place,in the image data in the three-dimensional space, the second analysisinformation of the designated range as the object in thethree-dimensional space.

The reception unit may accept a selection of one of a plurality of itemsof the first analysis information and accept a selection of one of aplurality of items of the second analysis information. The display unitmay display the selected item of the first analysis information and theselected item of the second analysis information on the display.

The reception unit may accept a selection of presence or absence of thedisplay of the first analysis information and the second analysisinformation.

The image processing system may further include a storage unit to storethe first analysis information and the second analysis information in astorage device. The storage unit may read out the image data, the firstanalysis information and the second analysis information from thestorage device. The display unit may display the image data, the firstanalysis information and the second analysis information that are readout by the storage unit on the display.

The storage unit may export the image data, the first analysisinformation and the second analysis information to the storage device ina predetermined file format. The storage unit may import the image data,the first analysis information and the second analysis information thatare in the predetermined format from the storage device.

According to still another embodiment, an information processingapparatus is provided, that displays image data whose angle is widerthan a predetermined angle and analysis information of the image data.The image processing apparatus includes an information associating unitto associate first analysis information obtained by analyzing the imagedata with the image data; a placing unit to place, in athree-dimensional space, the image data with which the first analysisinformation is associated; an analysis information placing unit to placesecond analysis information obtained by analyzing the image data as anobject in a three-dimensional space in association with the image datain the three-dimensional space; and a display unit to display the firstanalysis information, the second analysis information and the image dataon a display.

An image processing apparatus according to still another aspect of thepresent invention includes an information associating unit to associatefirst analysis information obtained by analyzing image data whose angleis wider than a predetermined angle with the image data; a placing unitto place, in a three-dimensional space, the image data with which thefirst analysis information is associated; an analysis informationplacing unit to place second analysis information obtained by analyzingthe image data as an object in a three-dimensional space in associationwith the image data in the three-dimensional space, wherein the imageprocessing apparatus transmits the first analysis information, thesecond analysis information, and the image data to another informationprocessing apparatus to cause the another information processingapparatus to display the transmitted information.

A program that causes an information processing apparatus that displaysimage data whose angle is wider than a predetermined angle and analysisinformation of the image data to function as: an information associatingunit to associate first analysis information obtained by analyzing theimage data with the image data; a placing unit to place, in athree-dimensional space, the image data with which the first analysisinformation is associated; an analysis information placing unit to placesecond analysis information obtained by analyzing the image data as anobject in a three-dimensional space in association with the image datain the three-dimensional space; and a display unit to display the firstanalysis information, the second analysis information and the image dataon a display.

A program that causes an information processing apparatus to execute amethod comprising: associating first analysis information obtained byanalyzing image data whose angle is wider than a predetermined anglewith the image data; a placing unit to place, in a three-dimensionalspace, the image data with which the first analysis information isassociated; an analysis information placing unit to place secondanalysis information obtained by analyzing the image data as an objectin a three-dimensional space in association with the image data in thethree-dimensional space, transmitting the first analysis information,the second analysis information, and the image data to anotherinformation processing apparatus to cause the another informationprocessing apparatus to display the transmitted information.

The above-described embodiments are illustrative and do not limit thepresent invention. Thus, numerous additional modifications andvariations are possible in light of the above teachings. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of the present invention.

Any one of the above-described operations may be performed in variousother ways, for example, in an order different from the one describedabove.

Each of the functions of the described embodiments may be implemented byone or more processing circuits or circuitry. Processing circuitryincludes a programmed processor, as a processor includes circuitry. Aprocessing circuit also includes devices such as an application specificintegrated circuit (ASIC), DSP (digital signal processor), FPGA (fieldprogrammable gate array) and conventional circuit components arranged toperform the recited functions.

What is claimed is:
 1. A communication terminal comprising: circuitryto: calculate, for each of a plurality of captured-image data items thatare obtained by continuous image capturing, a center point of an objectdetection area of the captured-image data item; render a heat map imagehaving a gradation of color, the gradation of color of the heat mapimage being determined based on weighting of colors of the heat mapimage with the calculated center points of the captured-image dataitems; map the heat map image on a captured-image represented by one ofthe plurality of captured-image data items; and display thecaptured-image on which the heat map image is mapped.
 2. Thecommunication terminal of claim 1, wherein the captured-image is afull-spherical panoramic image, and the circuitry is further configuredto control the display of full-spherical panoramic image such that theheat map image is placed on a front side of the full-spherical panoramicimage and a back surface of the heat map image is displayed.
 3. Thecommunication terminal of claim 1, further comprising: a memory to storea score table associating each of a plurality of types of the centerpoint with a score, wherein the circuitry is further configured tocalculate the center point based on the score table, and determineweighting of the colors of the heat map image such that the center pointwith a higher score is assigned with a darker color in the heat mapimage.
 4. The communication terminal of claim 3, wherein the pluralityof types of the center point includes each one of the center points, afirst average center point indicating a center point among a pluralityof the center points, a second average center point indicating a centerpoint among a plurality of the first average center points, the scoretable associates the second average center point with a highest score,the score table associates the first average center point with a secondhighest score, and the score table associates the center point with alowest score.
 5. The communication terminal of claim 3, wherein in acase in which a plurality of center points of the center points of theplurality of captured-image data items are present in a predeterminedrange, the circuitry is further configured to: integrate the pluralityof center points in the predetermined range into a single integratedcenter point; determine a sum of the scores of the plurality of centerpoints as a score of the single integrated center point; and weight thecolors of the heat map image using the single integrated center pointand the sum of the scores.
 6. The communication terminal of claim 1,wherein the circuitry is further configured to define the objectdetection area by an object position and an object area, the objectposition indicating a position of an arbitrary corner of a rectanglecontaining an object, and the object area indicating a width and aheight of the rectangle.
 7. The communication terminal of claim 1,wherein the communication terminal is one of a personal computer, asmart phone, a tablet, and a smartwatch.
 8. An image communicationsystem comprising: the communication terminal of claim 1; and an imagemanagement apparatus that manages the plurality of captured-image dataitems, wherein the communication terminal further includes a receiver toreceive the plurality of captured-image data items from the imagemanagement apparatus.
 9. A method for controlling display of capturedimages, comprising: calculating, for each of a plurality ofcaptured-image data items that are obtained by continuous imagecapturing, a center point of an object detection area of thecaptured-image data item; rendering a heat map image having a gradationof color, the gradation of color of the heat map image being determinedbased on weighting of colors of the heat map image with the calculatedcenter points of the captured-image data items; mapping the heat mapimage on a captured-image represented by one of the plurality ofcaptured-image data items; and displaying the captured-image on whichthe heat map image is mapped.
 10. A non-transitory computer-readablemedium including a computer-readable program, wherein thecomputer-readable program when executed on a computer causes thecomputer to perform a method, the method comprising: calculating, foreach of a plurality of captured-image data items that are obtained bycontinuous image capturing, a center point of an object detection areaof the captured-image data item; rendering a heat map image having agradation of color, the gradation of color of the heat map image beingdetermined based on weighting of colors of the heat map with thecalculated center points; mapping the heat map image on a captured-imagerepresented by one of the plurality of captured-image data items; anddisplaying the captured-image on which the heat map image is mapped.